Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

A Survey of Image-based Relighting Techniques

Biswarup Choudhury and Sharat Chandran

ViGILIIT Bombay

Mumbaiphone, email: biswarup,sharat@cse.iitb.ac.inwww: www.cse.iitb.ac.in/biswarup,sharat

Abstract

Image-based Relighting (IBRL) has recently attracteda lot of research interest for its ability to relight realobjects or scenes, from novel illuminations capturedin natural/synthetic environments. Complex lightingeffects such as subsurface scattering, interreflection,shadowing, mesostructural self-occlusion, refractionand other relevant phenomena can be generated usingIBRL. The main advantage of image-based graphics isthat the rendering time is independent of scene com-plexity as the rendering is actually a process of manip-ulating image pixels, instead of simulating light trans-port. The goal of this paper is to provide a completeand systematic overview of the research in ImagebasedRelighting. We observe that essentially all IBRL tech-niques can be broadly classified into three categories(Fig. 9), based on how the scene/illumination infor-mation is captured: Reflectance function-based, Basisfunction-based and Plenoptic function-based. We dis-cuss the characteristics of each of these categories andtheir representative methods. We also discuss aboutthe sampling density and types of light source(s), rele-

Digital Peer Publishing LicenceAny party may pass on this Work by electronicmeans and make it available for download underthe terms and conditions of the current versionof the Digital Peer Publishing Licence (DPPL).The text of the licence may be accessed andretrieved via Internet athttp://www.dipp.nrw.de/.First presented at the International Conferenceon Computer Graphics Theory and ApplicationsGRAPP 2006, extended and revisedfor JVRB

vant issues of IBRL.

Keywords: Image-based Relighting, Survey, Image-based Techniques, Augmented Reality.

1 Introduction

Image-based Modeling and Rendering (IBMR) syn-thesizes realistic images from pre-recorded imageswithout a complex and long rendering process as intraditional geometry-based computer graphics. Themajor drawback of IBMR is its inherent rigidity. MostIBMR techniques assume a static illumination condi-tion. Obviously, these assumptions cannot fully sat-isfy the computer graphics needs since illuminationmodification is a key operation in computer graph-ics. The ability to control illumination of the modeledscene enhances the three-dimensional illusion, whichin turn improves viewers’ understanding. If the illu-mination can be modified by relighting the images, in-stead of rendering the geometric models, the time forimage synthesis will be independent of the scene com-plexity. This also saves the artist/designer enormousamount of time in fine tuning the illumination con-ditions to achieve realistic atmospheres. Applicationsrange from global illumination and lighting design toaugmented and mixed reality, where real and virtualobjects are combined with consistent illumination.

Two major motivations for IBRL are :

• Allows the user to vary illuminance of the whole(or only interesting portions of the) scene improv-ing recognition and satisfaction.

• Brings us a step closer to realizing the use ofimage-based entities as the basic rendering prim-itives/ entities.

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

For didactic purposes, we classify image-basedrelighting techniques into three categories, namely:Reflectance-based, Basis Function-based, PlenopticFunction-based. These categories should be actuallyviewed as a continuum rather than absolute discreteones, since there are techniques that defy these strictcategorizations.

Reflectance Function-based Relighting tech-niques explicitly estimate the reflectance function ateach visible point of the object or scene. This is alsoknown as the Anisotropic Reflection Model [Kaj85] orthe Bidirectional Surface Scattering Reflectance Dis-tribution Function (BSSRDF) [JMLH01]. It is definedas the ratio of the outgoing to the incoming radiance.Reflectance estimation can be achieved with calibratedlight setup, which provide full control of the incidentillumination. A reflectance function is modeled withthe data of the scene, captured under varying illumina-tion and view direction. Techniques then apply novelillumination and use the reflectance function calcu-lated, for generating novel illumination effects in thescene.

Basis Function-based Relighting techniques takeadvantage of the linearity of the rendering operatorwith respect to illumination, for a fixed scene. Reren-dering is accomplished via linear combination of a setof pre-rendered “basis” images. These techniques, forthe purpose of computing a solution, determine a time-independent basis - a small number of “generative”global solutions that suffice to simulate the set of im-ages under varying illumination and viewpoint.

Plenoptic Function-based Relighting techniquesare based on the computational model, the PlenopticFunction [AB91]. The original plenoptic function ag-gregates all the illumination and the scene changingfactors in a single “time” parameter. So most researchconcentrates on view interpolation and leaves the timeparameter untouched (illumination and scene static).The plenoptic function-based relighting techniques ex-tract the illumination component from the aggregatetime parameter, and facilitate relighting of scenes.

The remainder of the paper is organized as follows:Section 2, Section 3 and Section 4 respectively discusseach of the three relighting categories, along with theirrepresentative methods. In Section 5, we discuss someof the other relevant issues of relighting. We then pro-vide some directions of future research in Section 6.Finally, we conclude with our final remarks in Section7.

2 Reflectance Function

A reflectance function is the measurement of how ma-terials reflect light, or more specifically, how theytransform incident illumination into radiant illumina-tion. The Bidirectional Reflectance Distribution Func-tion (BRDF) [NRH+77] is a general form of represent-ing surface reflectivity. A better representation is theBidirectional Surface Scattering Reflectance Distribu-tion Function (BSSRDF) [JMLH01], which model ef-fects such as color bleeding, translucency and dif-fusion of light across shadow boundaries, otherwiseimpossible with a BRDF model. As introduced by[DHT+00], the reflectance functionR, an 8D function,determines the light transfer between light entering abounding volume at a direction and position ψincident

and leaving at ψexitant:

R = R(ψincident, ψexitant)

This calculated reflectance function can be used tocompute relit images of the objects, lit with novel il-lumination. The computation for each relit pixel is re-duced to multiplying corresponding coefficients of thereflectance function and the incident illumination.

Lexitant(ω) =∫

ΩRLincident(ω)dω

where, Ω is the space of all light directions over ahemisphere centered around the object to be illumi-nated (ω ∈ Ω). For every viewing direction, each pixelin an image stores its appearance under all illuminationdirections. Thus each pixel in an image is a sample ofthe reflectance function. The reflectance functions aresampled from real objects by illuminating the objectfrom a set of directions while recording photographsfrom different viewpoints.

We classify the estimation of reflectance functionsinto three different categories: Forward, Inverse andPre-computed radiance transport.

2.1 Forward

The forward methods of estimating reflectance func-tions sample these functions exhaustively and tabu-late the results. For each incident illumination, theystore the reflectance function weights for a fixed ob-served direction. The forward method of estimatingreflectance functions can further be divided into twocategories, on the basis of illumination informationprovided, Known and Unknown.

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

Figure 1: Mirrored balls, representing illumination of an environment, used for relighting faces [DHT+00].Image cite from the website http://www.debevec.org/Research/LS/ with permission from Mr. Debevec.

Figure 2: An arrangement of chess pieces, illuminated by different types of incident light fields [MPDW04].Image cite from the website http://www.cs.kuleuven.be/ graphics/CGRG.PUBLICATIONS/RW4DILF/ withpermission from Mr. Dutre.

The techniques with known illumination incorpo-rate the information in their setup. The user is pro-vided full control over the direction, position and typeof incident illumination. This information is directlyused for finding the reflectance properties of the scene.[DHT+00] use the highest resolution incident illumi-nation with roughly 2000 directions and construct a re-flectance function for each observed image pixel fromits values over the space of illumination directions(Fig. 1).[MPDW04] sample the reflectance functionsfrom real objects by illuminating the object from a setof directions while recording the photographs. Theyreconstruct a smooth and continuous reflectance func-tion, from the sampled reflectance functions, using themultilevel B-spline technique. [MPDW04] exploit therichness in the angular and spatial variation of the inci-dent illumination, and measure six-dimensional slicesof the eight-dimensional reflectance field, for a fixedviewpoint (Fig. 2). On the other hand, [MGW01]store the coefficients of a biquadratic polynomial foreach texel, thereby improving upon the compactnessof the representation, and uses it to reconstruct the sur-face color under varied illumination conditions. Thereflectance functions and illumination are expressed

by a set of common basis functions, enabling a signif-icant speed-up in the relighting process. This methodmaintains all the high-frequency features such as high-lights and self-shadowing. [GLL+04] captures theeffects of translucency by illuminating individual sur-face points of a translucent object and measuring theimpulse response of the object in each case.

[WHON97, WHF01] propose a concept ofapparent-BRDF to represent the outgoing radiancedistribution passing through the pixel window on theimage plane. By treating each image as an ordinarysurface element, the radiance distribution of the pixelunder various illumination conditions is recorded ina table (Fig. 3). [KBMK01] samples the surfacesincident field to reconstruct a non-parametric apparentBRDF at each visible point on the surface.[BG01] it-eratively produces an approximation of the reflectancemodel of diffuse, specular, isotropic or anisotropictextured objects using a single image and the 3Dgeometric model of the scene.

Techniques with unknown incident illumination in-formation estimate the information. [NN04] use eyesof a human subject and compute a large field ofview of the illumination distribution of the environ-

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

Figure 3: Attic relighting using conventional light sources(top) and 5 spot lights(bottom) [WHF01]. Originalimage source: Tien-Tsin Wong, Pheng-Ann Heng, and Chi-Wing Fu, Interactive relighting of panoramas, IEEEComput. Graph. Appl. 21(2001), no. 2, 32-41. c©2001 IEEE

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

Figure 4: Using the mathematical framework of the imaging system of the eye, one can estimate the illuminationinformation [NN04]. Image cited from the website http://www1.cs.columbia.edu/CAVE/projects/world eye/with permission from Mr. Nayar.

Figure 5: Four relighting examples(top row) as linear combination of images, the coefficients being definedby novel images of a probe object(bottom, left image) which are reconstructed with the sampled probe im-ages(right) [FBS05]. Original image source: Martin Fuchs, Volker Blanz, and Hans-Peter Seidel, Bayesianrelighting, Proceedings of the Eurographics Symposium on Rendering Techniques on July29th - July 1st, 2005,Eurographics Association, 2005, ISBN 3-905673-23-1, pp. 157-164. c©Eurographics Association 2005;Reproduced by kind permission of the Eurographics Association

Figure 6: A transparent dragon and a torus digitally composed onto background images, preserving the illumi-nation effects of refraction, reflection and colored filtering [ZWCS99],[CZH+00]

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

ment surrounding a person, using the characteristicsof the imaging system formed by the cornea of aneye and a camera viewing it (Fig. 4). Their assump-tion of a human subject in the scene, at all times,may not be practical though.[LKG+03] used six steelspheres to recover the light source positions. Theyfit an average BRDF function to the different mate-rials of the objects in the scene. Some other tech-niques [FBS05, TSED04] indirectly compute the inci-dent illumination information by using black snookerball/nonmetallic sphere (Fig. 5). A very early workof IBRL, Inverse Rendering [Mar98], solves for un-known lighting and reflectance properties of a scene,for relighting purposes.

2.2 Inverse

The inverse problem of estimation of reflectance func-tions can be be stated as follows: Given an observa-tion, what are the weights and parameters of the basisfunctions that best explain the observation?

Inverse methods observe an output and compute theprobability that it came from a particular region in theincident illumination domain. The incident illumina-tion is typically represented by a bounded region, suchas an environment map, which is modeled as a sumof basis functions [rectangular [ZWCS99] or Gaus-sian kernels [CZH+00]. They capture an environmentmatte, which in addition to capturing the foregroundobject and its traditional matte, also describes howthe object refracts and reflects light. This can thenbe placed in a new environment, where it will refractand reflect light from that scene (Fig. 6). Techniques[MLP04, PD03] have been proposed which progres-sively refine the approximation of the reflectance func-tion with an increasing number of samples.

For a more accurate reflectance estimation,[MPZ+02] combine a forward method [DHT+00] forthe low-frequency surface reflectance function and aninverse method, environment matting [CZH+00], forthe high-frequency surface reflectance function. Thisis used for capturing all the complex lighting effects,like high-frequency reflections and refractions.

2.3 Pre-computed Radiance transport

A global transport simulator creates functions over theobject’s surface, representing transfer of arbitrary inci-dent lighting, into transferred radiance which includesglobal effects like shadows, self-interreflections, oc-clusion and scattering effects. When the actual light-

ing condition is substituted at run-time, the resultingmodel provides global illumination effects.

The radiance transport is pre-computed using a de-tailed model of the scene [SKS02]. To improveupon the rendering performance, the incident illumi-nation can be represented using spherical harmon-ics [KSS02, RH01, SKS02] or wavelets [NRH03].The reflectance field, stored per vertex as a transfermatrix, can be compressed using PCA [SHHS03] orwavelets [NRH03].

[NRH04] focuses on relighting for changing illumi-nation and viewpoint, while including all-frequencyshadows, reflections and lighting. They propose anovel triple product integrals based technique of fac-torizing the visibility and the material properties. Re-cently, [WTL05] presented a method of relightingtranslucent objects under all-frequency lighting. Theyapply a two-pass hierarchical technique for computingnon-linearly approximated transport vectors due to dif-fuse multiple scattering.

3 Basis Function

Basis Function based techniques decompose the lumi-nous intensity distributions into a series of basis func-tions, and illuminances are obtained by simply sum-ming luminance from each light source whose lumi-nous intensity distribution obey each basis function.Assuming multiple light sources, luminance at a cer-tain point is obtained by calculating the luminancefrom each light source and summing them. In general,luminance calculation obeys the two following rules ofsuperposition:

1. The image resulting from an additive combina-tion of two illuminants is just the sum of the im-ages resulting from each of the illuminations in-dependently.

2. Multiplying the intensity of the illuminationsources by a factor of α results in a rendered im-age that is multiplied by the same factor α.

These techniques calculate luminance in the case ofalterations in the luminous distributions and the direc-tion of light sources. The luminous intensity distri-bution of a point light source is expressed as the sumof a series of basis distributions. Luminance due tolight source, whose luminance intensity distributioncorresponds to one of the basis distributions, is calcu-lated in advance and stored as basis luminance. Using

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

Figure 7: Illuminating faces using Illumination cones. First row (a): Input dataset, Second row (b): BasisImages, Third and Fourth row (d): Relit Faces [GKB98, GBK01]. Original image source: Athinodoros S.Georghiades, Peter N. Belhumeur, and David J. Kriegman, From few to many: Illumination cone models forface recognition under variable lighting and pose, IEEE Trans. Pattern Anal. Mach. Intell. 23(2001), no. 6,643-660. c©2001 IEEE

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

Figure 8: The tri-planar configuration and the dual light slab parametrization. Each captured value representsthe radiance reflected through (s, t) and received at a certain (u, v) when the scene is illuminated by a pointsource positioned at (q, r) [LWS02]

the aforementioned property 1, the luminance due tothe light source with luminous intensity distribution,is calculated by summing the pre-calculated basis lu-minances corresponding to each individual basis dis-tribution. Using property 2, the luminance due to alight source, whose luminous intensity distribution canbe expressed as the weighted sum of the basis distri-butions, is obtained by multiplying each basis lumi-nance with corresponding weights and summing them.Thus, once the basis luminance is calculated in thepre-process, the resulting luminance can be obtainedquickly by calculating the weighted sum of the basisluminances. Some desirable properties of a basis setof illumination functions are:

1. The basis functions should be general enough toform any light source one desires.

2. The number of basis functions should be small,since this corresponds to the number of basis im-ages we must actually store and render.

We classify the type of basis functions (used in Re-lighting) into five categories and provide their corre-sponding representative methods:

1. Steerable Functions [NSD94]: Under natural il-luminants like the sun, whose direction variesalong a path on the sphere, a desirable propertyof the basis is to be rotation invariant. A steerablefunction is one which can be written as a linearcombination of rotated versions of itself.

2. Spherical Harmonics [DKNY95]: Spherical har-monics is an orthonormal basis defined over aunit sphere. In the context of interior light-ing design, spherical harmonics can express lightsources from different directions. They are effec-tive for recovering illumination that is well local-ized in the frequency domain.

3. Singular Value Decomposition/Principal Compo-nent Analysis [GBK01] (Fig. 7), [GKB98, OK01,HWT+04]: These techniques are used for sim-plifying a dataset, or specifically reduce the di-mensionality of a data set while retaining as muchinformation as is possible. They compute a com-pact and optimal description of the data set.

4. N-mode SVD: Multilinear algebra of higherorderTensors [VT04, FKIS02, SVLD03, TZL+02]: Itis an extension to tensors, of the conventional ma-trix singular value decomposition. A major ad-vantage of this model is that it is purely image-based, thereby avoiding the nontrivial problem of3D geometric structure from the image data. Fur-ther, this formulation can disentangle and explic-itly represent each of the multiple factors inher-ent to image formationillumination change, view-point change etc.

4 Plenoptic Function

The appearance of the world can be thought of as thedense array of light rays filling the space, which can beobserved by posing eyes or cameras in space. These

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

light rays can be represented through the plenopticfunction (from plenus, complete or full; and optics)[AB91]. As pointed by Adelson and Bergen:

The world is made of three-dimensional objects, butthese objects do not communicate their properties di-rectly to an observer. Rather, the objects fill the spacearound them with the pattern of light rays that con-stitutes the plenoptic function, and the observer takessamples from this function. The plenoptic functionserves as the sole communication link between thephysical objects and their retinal images. It is the in-termediary between the world and the eye.

All basic visual measurements can be considered tocharacterize local change along one or more dimen-sions of a single function that describes the structure ofthe information in the light impinging on an observer.The plenoptic function represents the intensity of thelight rays passing through the camera center at everylocation, at every possible viewing angle. The plenop-tic function is a 7D function that models a 3D dy-namic environment by recording the light rays at everyspace location (Vx, Vy, Vz), from every possible direc-tion (θ, φ), over any range of wavelengths (λ) and atany time (t), i.e.,

P = P (7)(Vx, Vy, Vz, θ, φ, λ, t)

An image of a scene with a pinhole camera recordsthe light rays passing through the cameras center-ofprojection. They can also be considered as samplesof the plenoptic function. Basically, the function tellsus how the environment looks when our eye is posi-tioned at V = (Vx, Vy, Vz). The time parameter t ac-tually models all the other unmentioned factors suchas the change of illumination and the scene.

Plenoptic Function-based relighting techniques pro-pose new formulations of the plenoptic function,which explicitly specify the illumination component.Using these formulations, one can generate complexlighting effects. One can simulate various lightingconfigurations such as multiple light sources, lightsources with different colors and also arbitrary typesof light sources (Section 5.1).

4.1 Representative Techniques

[WH04] discuss a new formulation of the plenopticfunction, Plenoptic Illumination Function, which ex-plicitly specifies the illumination component. Theypropose a local illumination model, which utilizesthe rules of superposition for relighting under various

lighting configurations. [LWS02] on the other hand,propose a representation of the plenoptic function, thereflected irradiance field. The reflected irradiance fieldstores the reflection of surface irradiance as an illumi-nating point light source moves on a plane. With thereflected irradiance field, the relit object/scene can besynthesized simply by interpolating and superimpos-ing appropriate sample reflections (Fig. 8).

5 Discussion

In this section, we discuss some of the relevant issuesinvolving IBRL. We also provide a brief comparisonbetween all the categories of IBRL techniques.

5.1 Light Source type

Illumination is a complex and high-dimensional func-tion of computer graphics. To reduce the dimensional-ity and to analyze their complexity and practicality, itis necessary to assume a specific type of light source.Two types of light sources most commonly used are:

1. Directional Light Source (DLS): A DLS emitsparallel rays which do not diverge or becomedimmer with distance. It is parametrized usingonly two variables (θ, φ), which denotes the di-rection of the light vector. For planar surfaceslighted by a DLS, the degree of shading will bethe same right across the surface. The compu-tations required for directional lights are there-fore considerably less. Using a DLS is also moremeaningful, because the captured pixel value inan image tells us what the surface elements be-hind the pixel window actually look like, whenall surface elements are illuminated by parallelrays in the direction of the viewing point. DLSserves well with synthetic object/scene where itis used to approximate the light coming from anextremely distant light source. But it poses practi-cal difficulties for capturing real and large object/scene. They can be approximated with strongspotlights at a distance which greatly exceeds thesize of the object/scene.

2. Point Light Source (PLS): A PLS shines uni-formly in all directions. Its intensity decreaseswith the distance to the light source. A PLS isparametrized using three variables (Px, Py, P z),which denote the 3D position of the PLS in space.As a result, the angle between the light source

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

Figure 9: Pictorial representation of IBRL categorization.

and the normals of the various affected surfacescan change dramatically from one surface to thenext. In the presence of multiple light sources,this means that for every vertex, one has to de-termine the direction of the light vector corre-sponding to a light source. This requires deter-mination of the depth map of the images usingcomputer vision algorithms, which though pro-vide good approximations, make the lighting cal-culations computationally intensive. Point lightsource are usually close to the observer and somore practical for real and large objects/scenes.

5.2 Sampling

Sampling is one of the key issues of image-basedgraphics. It is a non-trivial problem because it involvesthe complex relationship among three elements: thedepth and texture of the scene, the number of sam-ple images, and the rendering solution. One needs todetermine the minimum sampling rate for anti-aliasedimage-based rendering. Comparatively, very little re-search [CCST00, SK00, ZC04, ZC01, ZC03] has goneinto trying to tackle this problem.

In the context of IBRL, sampling deals with theillumination component for efficient and realistic re-lighting [WH04]. [LWS02] prove that there exists ageometry-independent bound of the sampling interval,which is analytically bound to the BRDF of the scene.It ensures that the intensity error in the relit image issmaller than a user-specified tolerance, thus eliminat-

ing noticeable artifacts.

5.3 Comparison

We compare all the IBRL categories (Fig. 9) on thebasis of the following factors:

• Object and Image Space: Reflectance function-based relighting techniques are closest to the ob-ject space since they try to estimate a scene/objectproperty, reflectance. The essence of these tech-niques lie in modeling the reflectance of thescene/object in a correct and efficient mannerfrom the moderately sampled images of the same(and not in densely sampling it). The model ofthe reflectance if computed correctly, can repro-duce very realistic illumination effects, but in-correct modeling can result in unacceptable ar-tifacts. Since the scene/object needs to be onlymoderately sampled, hence the acquisition timefor these techniques are usually less as comparedto the two other IBRL categories. A major dis-advantage of reflectance function-based relight-ing techniques is that not all illumination effectscan be reproduced using a single model of thereflectance. So, a scene composed of objectswith different reflectance cannot all be realisti-cally relit. Once the compact reflectance modelis computed, relighting can be performed in re-altime. Some reflectance function-based tech-niques (precomputed radiance transport), unlike

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

the other two IBRL categories, require geometryof the scene/object to estimate the reflectance ofthe same. Though these techniques produce re-sults devoid of artifacts, the geometry of the ob-ject/ scene may not always be available.

Plenoptic function-based relighting techniquesare closest to the image space. They sample thescene under different illumination conditions anduse these images for relighting. Since these tech-niques are entirely image-dependent techniques,to create interesting and complex illumination ef-fects (specularity, highlights, shadows, refractionetc.), they rely heavily on sampling the object/scene densely. Therefore in general, plenopticfunction-based techniques have a huge datasetto reproduce all the desired illumination effects.Therefore, the acquisition (photographs of a realscene/object) or synthesis (images of an artifi-cial scene/object) of the huge dataset is a longand tedious process. Processing of this hugedataset to create the desired illumination effectsalso demand huge computational burden and highmemory requirements. The main advantage ofplenoptic function-based techniques is that theyare measurement-based, and so they produce veryrealistic relighting.

While reflection-based relighting and plenopticfunction-based relighting techniques are in gen-eral at the two extremes of object and imagespace respectively, basis function-based relight-ing techniques are a confluence of the two spaces.Basis function-based techniques sample the scenedensely and then create a compact intermediaterepresentation of the scene/object for further pro-cessing and relighting. Since these techniquesalso depend on the sampling, hence acquisition(or synthesis) of the dataset and computing thebasis representation takes a considerable amountof time. Further, more importantly, processingthe input images into their intermediate represen-tation is dependent on the effectiveness and ap-propriateness of the basis function used and so isprone to errors creeping in and resulting in un-desirable illumination effects. Relighting can beachieved in realtime due to the compact basis rep-resentation of the scene/object.

• Frequency components: While both reflectancefunction and plenoptic function-based relightingtechniques can capture both the high and low

frequency components, basis function-based re-lighting techniques are usually capable of cap-turing only the low frequency effects. For-ward reflectance function-based techniques cap-ture the low frequency components and the in-verse methods capture the high frequency com-ponents. Plenoptic function-based techniques onthe other hand, sample the scene densely in thepresence of high frequency components and soare able to realistically reproduce relighting.

• Sampling density: For the purposes of IBRL,the sampling density of the corresponding tech-niques usually increases in the following order -reflectance function-based, basis function-based,plenoptic function-based. Therefore, the acquisi-tion, processing and relighting time also increasein the same order.

• Applications: Reflectance-based relighting tech-niques can be utilized to edit the surface prop-erties of an object in a scene (for example, con-verting a diffuse object to specular) in additionto performing IBRL. The techniques of the othertwo categories are not capable of generating sucheffects. They mainly illuminate the scene witha novel illumination with or without change inviewpoint. So reflectance function-based IBRLcould be efficiently utilized for applications thatrequire editing both lighting configurations andmaterial properties of the scene/objects in thescene.

6 Future directions

A lot of research remains to be done in IBRL. Someideas being:

1. Efficient Representation: BRDF function-basedIBRL techniques require huge number of sam-ples to accurately estimate a reflectance function.Most techniques, for practical purposes, considerlow-frequency components, which compromiseswith the visual quality of the rendered image. Al-most all basis function-based techniques also re-quire a number of images for relighting. Thus, alot of research is required to find accurate and ef-ficient representations of a scene, which captureall the complex phenomenas of lighting and re-flectance functions. Variability in terms of view-point and illumination leads to huge data sets,

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

which further incurs huge computational costs. Arelated area which deserves considerable investi-gation, is IBRL for real and large environments.

2. Sampling: Most techniques do not deal withthe minimum sampling density required foranti-aliased IBRL. [LWS02] discuss about ageometry-independent sampling density based onradiometric tolerance. Though this serves ourpurpose of efficient sampling of certain scenes,what we need is a photometric tolerance, whichtakes into account the response function of hu-man vision. [DMZ+05] discuss the importanceof psychophysical quality scale for realistic IBRLof glossy surfaces.

3. Compression: No matter how much the storageand memory increase in the future, compressionis always useful to keep the IBRL data at a man-ageable size. A high compression ratio in IBRLrelies heavily on how good the images can bepredicted. The sampled images for IBRL, usu-ally have a strong inter-pixel and intra-pixel cor-relation, which needs to be harnessed for effi-cient compression. Currently, techniques such asspherical harmonics, vector quantization, directcosine transform and spherical wavelets are usedfor compressing the datasets of IBRL, but all ofthese have their own inherent disadvantages.

4. Dynamics: Most IBRL techniques deal withstatic environments, in terms of change in geom-etry of the scene/object. With the development ofhigh-end programmable graphics processors, it isconceivable that IBRL can be applied to dynamicenvironments.

7 Final remarks

We have surveyed all the recent developments in thefield of Image-based Relighting and, in particular,classified them into three categories based on howthey capture the scene/illumination information: Re-flectance Function-based, Basis Function-based andPlenoptic Function-based. We have presented each ofthe categories along with their corresponding repre-sentative methods. Relevant issues of IBRL like typeof light source and sampling have also been discussed.

It is interesting to note the trade-off between geom-etry and images, needed for anti-aliased image-basedrendering. Efficient representation, realistic rendering,

limitations of computer vision algorithms and compu-tational costs should motivate researchers to invent ef-ficient image-based relighting techniques in future.

8 Acknowledgments

We would like to thank the members of ViGIL, IITBombay for their continued help and support.

This research was supported by the fellowship fundendowed by Infosys Technologies Limited, India.

References

[AB91] Edward H. Adelson and James R.Bergen, The plenoptic function and theelements of early vision, pp. 3–20, MITPress, 1991, ISBN 0-262-12155-7.

[BG01] Samuel Boivin and Andre Gagalowicz,Image-based rendering of diffuse, spec-ular and glossy surfaces from a sin-gle image, SIGGRAPH 01: Proceedingsof the 28th annual conference on Com-puter graphics and interactive techniques,ACM Press, 2001, ISBN 1-58113-292-1,pp. 107–116.

[CCST00] Jin-Xiang Chai, Shing-Chow Chan,Heung-Yeung Shum, and Xin Tong,Plenoptic sampling, SIGGRAPH’00: Proceedings of the 27th annualconference on Computer graphicsand interactive techniques, ACMPress/Addison-Wesley Publishing Co.,2000, ISBN 1-58113-208-5, pp. 307–318.

[CZH+00] Yung-Yu Chuang, Douglas E. Zongker,Joel Hindorff, Brian Curless, David H.Salesin, and Richard Szeliski, Environ-ment matting extensions: towards higheraccuracy and real-time capture, SIG-GRAPH ’00: Proceedings of the 27thannual conference on Computer graph-ics and interactive techniques, ACMPress/Addison-Wesley Publishing Co.,2000, ISBN 1-58113-208-5, pp. 121–130.

[DHT+00] Paul Debevec, Tim Hawkins, ChrisTchou, Haarm-Pieter Duiker, WestleySarokin, and Mark Sagar, Acquiring the

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

reflectance field of a human face, SIG-GRAPH ’00: Proceedings of the 27thannual conference on Computer graph-ics and interactive techniques, ACMPress/Addison-Wesley Publishing Co.,2000, ISBN 1-58113-208-5, pp. 145–156.

[DKNY95] Yoshinori Dobashi, Kazufumi Kaneda,Hideki Nakatani, and Hideo Yamashita,A quick rendering method using basisfunctions for interactive lighting design,Computer Graphics Forum 14 (1995),no. 3, 229–240, ISSN 0167-7055.

[DMZ+05] Olivier Dumont, Vincent Masselus, Pe-ter Zaenen, Johan Wagemans, and PhilipDutre, A perceptual quality scale forimage-based relighting of glossy sur-faces, Tech. Report CW417, KatholiekeUniversiteit Leuven, June 2005.

[FBS05] Martin Fuchs, Volker Blanz, and Hans-Peter Seidel, Bayesian relighting, Pro-ceedings of the Eurographics Sympo-sium on Rendering Techniques on July29th - July 1st, 2005, Eurographics As-sociation, 2005, ISBN 3-905673-23-1,pp. 157–164.

[FKIS02] Ryo Furukawa, Hiroshi Kawasaki, Kat-sushi Ikeuchi, and Masao Sakauchi,Appearance based object modeling us-ing texture database: acquisition, com-pression and rendering, EGRW ’02:Proceedings of the 13th Eurographicsworkshop on Rendering (Aire-la-Ville,Switzerland), Eurographics Association,2002, ISBN 1-58113-534-3, pp. 257–266.

[GBK01] Athinodoros S. Georghiades, Peter N.Belhumeur, and David J. Kriegman,From few to many: Illumination conemodels for face recognition under vari-able lighting and pose, IEEE Trans. Pat-tern Anal. Mach. Intell. 23 (2001), no. 6,643–660, ISSN 0162-8828.

[GKB98] A. S. Georghiades, D. J. Kriegman, ,and P. N. Belhumeur, Illumination conesfor recognition under variable lighting:Faces, CVPR 98: Proceedings of theIEEE Computer Society Conference on

Computer Vision and Pattern Recogni-tion, IEEE Computer Society0, 1998,ISBN 1063-6919, pp. 52–58.

[GLL+04] Michael Goesele, Hendrik P. A. Lensch,Jochen Lang, Christian Fuchs, and Hans-Peter Seidel, Disco: acquisition oftranslucent objects, ACM Trans. Graph.23 (2004), no. 3, 835–844, ISSN 0730-0301.

[HWT+04] Tim Hawkins, Andreas Wenger, ChrisTchou, Andrew Gardner, FredrikGoransson, and Paul E. Debevec,Animatable facial reflectance fields,Proceedings of the 15th EurographicsWorkshop on Rendering Techniques,2004, ISBN 3-905673-12-6, pp. 309–321.

[JMLH01] Henrik Wann Jensen, Stephen R.Marschner, Marc Levoy, and Pat Han-rahan, A practical model for subsurfacelight transport, SIGGRAPH 01: Pro-ceedings of the 28th annual conferenceon Computer graphics and interactivetechniques, ACM Press, 2001, ISBN

1-58113-292-1, pp. 511–518.

[Kaj85] James T. Kajiya, Anisotropic reflectionmodels, SIGGRAPH 85: Proceedingsof the 12th annual conference on Com-puter graphics and interactive techniques,ACM Press, 1985, ISBN 0097-8930,pp. 15–21.

[KBMK01] Melissa L. Koudelka, Peter N. Bel-humeur, Sebastian Magda, and David J.Kriegman, Image-based modeling andrendering of surfaces with arbitrarybrdfs, In Proceedings of Computer Vi-sion and Pattern Recognition CVPR01,2001, ISBN 0-7695-1271-0, pp. 568–575.

[KSS02] Jan Kautz, Peter-Pike Sloan, and JohnSnyder, Fast, arbitrary brdf shading forlow-frequency lighting using sphericalharmonics, EGRW ’02: Proceedings ofthe 13th Eurographics workshop on Ren-dering (Aire-la-Ville, Switzerland), Eu-rographics Association, 2002, ISBN 1-58113-534-3, pp. 291–296.

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

[LKG+03] Hendrik P. A. Lensch, Jan Kautz,Michael Goesele, Wolfgang Heidrich,and Hans-Peter Seidel, Image-based re-construction of spatial appearance andgeometric detail, ACM Trans. Graph. 22(2003), no. 2, 234–257, ISSN 0730-0301.

[LWS02] Zhouchen Lin, Tien-Tsin Wong, andHeung-Yeung Shum, Relighting with thereflected irradiance field: Representa-tion, sampling and reconstruction, Int. J.Comput. Vision 49 (2002), no. 2-3, 229–246, ISSN 0920-5691.

[Mar98] Stephen Robert Marschner, Inverse ren-dering for computer graphics, Ph.D. the-sis, Cornell University, 1998, AdviserDonald P. Greenberg.

[MGW01] Tom Malzbender, Dan Gelb, and HansWolters, Polynomial texture maps, SIG-GRAPH ’01: Proceedings of the 28thannual conference on Computer graph-ics and interactive techniques, CM Press,2001, ISBN 1-58113-292-1, pp. 519–528.

[MLP04] Wojciech Matusik, Matthew Loper, andHanspeter Pfister, Progressively-refinedreflectance functions from natural illu-mination, Proceedings of the 15th Euro-graphics Workshop on Rendering Tech-niques, Eurographics Association, 2004,ISBN 3-905673-12-6, pp. 299–308.

[MPDW04] Vincent Masselus, Pieter Peers, PhilipDutre, and Yves D. Willems, Smooth re-construction and compact representationof reflectance functions for image-basedrelighting, Proceedings of the 15th Eu-rographics Workshop on Rendering, Eu-rographics Association, 2004, ISBN 3-905673-12-6, pp. 287–298.

[MPZ+02] Wojciech Matusik, Hanspeter Pfister,Remo Ziegler, Addy Ngan, and LeonardMcMillan, Acquisition and renderingof transparent and refractive objects,EGRW ’02: Proceedings of the 13thEurographics workshop on Rendering(Aire-la-Ville, Switzerland), Eurograph-ics Association, 2002, ISBN 1-58113-534-3, pp. 267–278.

[NN04] Ko Nishino and Shree K. Nayar, Eyesfor relighting, ACM Trans. Graph. 23(2004), no. 3, 704–711, ISSN 0730-0301.

[NRH+77] F.E. Nicodemus, J.C. Richmond, J. J.Hsia, I. W. Ginsberg, and T. Limperis,Geometric considerations and nomen-clature for reflectance, NBS Monograph160, National Bureau of Standards (US),1977.

[NRH03] Ren Ng, Ravi Ramamoorthi, and PatHanrahan, All-frequency shadows usingnon-linear wavelet lighting approxima-tion, ACM Trans. Graph. 22 (2003),no. 3, 376–381, ISSN 0730-0301.

[NRH04] R. Ng, Ravi Ramamoorthi, and Pat Han-rahan, Triple product wavelet integralsfor all-frequency relighting, ACM Trans.Graph. 23 (2004), no. 3, 477–487, ISSN

0730-0301.

[NSD94] Jeffry S. Nimeroff, Eero Simoncelli,and Julie Dorsey, Efficient Re-renderingof Naturally Illuminated Environments,Fifth Eurographics Workshop on Ren-dering (Darmstadt, Germany), Springer-Verlag, 1994, ISBN 3-540-58475-7,pp. 359–373.

[OK01] Margarita Osadchy and Daniel Keren,Image detection under varying illumina-tion and pose, Proceedings of the EighthIEEE International Conference On Com-puter Vision ICCV’01, IEEE, 2001, ISBN

0-7695-1143-0, pp. 668–673.

[PD03] Pieter Peers and Philip Dutre, Waveletenvironment matting, EGRW ’03:Proceedings of the 14th Eurograph-ics workshop on Rendering (Aire-la-Ville,Switzerland), EurographicsAssociation, 2003, ISBN 3-905673-03-7,pp. 157–166.

[RH01] Ravi Ramamoorthi and Pat Hanrahan, Anefficient representation for irradiance en-vironment maps, SIGGRAPH ’01: Pro-ceedings of the 28th annual conferenceon Computer graphics and interactivetechniques, ACM Press, 2001, ISBN 1-58113-374-x, pp. 497–500.

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

[SHHS03] Peter-Pike Sloan, Jesse Hall, John Hart,and John Snyder, Clustered principalcomponents for precomputed radiancetransfer, ACM Trans. Graph. 22 (2003),no. 3, 382–391, ISSN 0730-0301.

[SK00] Heung-Yeung Shum and Sing BingKang, A review of image-based renderingtechniques, IEEE/SPIE Visual Commu-nications and Image Processing (VCIP),EEE/SPIE, 2000, ISBN 0-8194-3703-4,pp. 2–13.

[SKS02] Peter-Pike Sloan, Jan Kautz, and JohnSnyder, Precomputed radiance transferfor real-time rendering in dynamic,lowfrequency lighting environments,SIGGRAPH ’02: Proceedings of the29th annual conference on Computergraphics and interactive techniques,ACM Press, 2002, ISBN 1-58113-521-1,pp. 527–536.

[SVLD03] F. Suykens, K. Vom, A. Lagae, andP. Dutre, Interactive rendering with bidi-rectional texture functions, ComputerGraphics Forum 22 (2003), no. 3, 463–472, ISSN 1467-8659.

[TSED04] Chris Tchou, Jessi Stumpfel, Per Einars-son, and Paul Debevec, Unlighting theparthenon, SIGGRAPH 2004 Sketch,2004.

[TZL+02] Xin Tong, Jingdan Zhang, Ligang Liu,Xi Wang, Baining Guo, and Heung-Yeung Shum, Synthesis of bidirectionaltexture functions on arbitrary surfaces,SIGGRAPH ’02: Proceedings of the 29thannual conference on Computer graphicsand interactive techniques, ACM Press,2002, ISBN 1-58113-521-1, pp. 665–672.

[VT04] M. Alex O. Vasilescu and DemetriTerzopoulos, Tensortextures: multilin-ear image-based rendering, ACM Trans.Graph. 23 (2004), no. 4, 336–342, ISSN

0730-0301.

[WH04] Tien-Tsin Wong and Pheng-Ann Heng,Image-based relighting: representation

and compression, The Kluwer Interna-tional Series in Engineering and Com-puter Science, pp. 161–180, Kluwer Aca-demic Publishers, 2004, ISBN 1-4020-7774-2.

[WHF01] Tien-Tsin Wong, Pheng-Ann Heng, andChi-Wing Fu, Interactive relighting ofpanoramas, IEEE Comput. Graph. Appl.21 (2001), no. 2, 32–41, ISSN 0272-1716.

[WHON97] Tien-Tsin Wong, Pheng-Ann Heng, Siu-Hang Or, and Wai-Yin Ng, Image-based rendering with controllable illu-mination, Proceedings of the Eurograph-ics Workshop on Rendering Techniques’97, Springer-Verlag, 1997, ISBN 3-21-83001-4, pp. 13–22.

[WTL05] Rui Wang, John Tran, and David Lue-bke, All-frequency interactive relightingof translucent objects with single andmultiple scattering, ACM Trans. Graph.24 (2005), no. 3, 1202–1207, ISSN 0730-0301.

[ZC01] C. Zhang and T. Chen, Generalizedplenoptic sampling, Tech. ReportAMP01-06, Carnegie Mellon TechnicalReport, 2001.

[ZC03] Chen Zhang and T. Chen, Spectral anal-ysis for sampling image-based render-ing data, IEEE Transactions on Circuitsand Systems for Video Technology 13(2003), no. 11, 1038–1050, ISSN 1051-8215.

[ZC04] C. Zhang and T. Chen, A survey onimage-based rendering: Representation,sampling and compression, SPIC 19(2004), no. 1, 1–28.

[ZWCS99] Douglas E. Zongker, Dawn M. Werner,Brian Curless, and David H. Salesin,Environment matting and compositing,SIGGRAPH ’99: Proceedings of the 26thannual conference on Computer graph-ics and interactive techniques, ACMPress/Addison-Wesley Publishing Co.,1999, ISBN 0-201-48560-5, pp. 205–214.

urn:nbn:de:0009-6-21208, ISSN 1860-2037

Journal of Virtual Reality and Broadcasting, Volume 4(2007), no. 7

CitationBiswarup Choudhury and Sharat Chandran, A Sur-vey of Image-based Relighting Techniques, Journalof Virtual Reality and Broadcasting, 4(2007),no. 7, January 2009, urn:nbn:de:0009-6-21208,ISSN 1860-2037.

urn:nbn:de:0009-6-21208, ISSN 1860-2037