Image-Based Material Editing

Erum Khan Erik ReinhardUniversity of Central Florida

Roland Fleming Heinrich B̈ulthoffMax Planck Institute for Biological Cybernetics

Abstract

Photo editing software allows digital images to be blurred, warpedor re-colored. However, it is not currently possible to change thematerial appearance of an object except by painstakingly paintingover the appropriate pixels. We present a set of methods for au-tomatically replacing one material with another material, startingwith only a single high dynamic range (HDR) image as input.

1 Objectives

We address the problem of editing a photograph of an object—say abronze sculpture or a porcelain vase—so that it appears to be madeout of some completely different material, such as glass, plastic, orwax. To do this in a physically correct way would involve accu-rately estimating the lighting, material properties and 3D geometryof the object from the image; a task that even state-of-the-art com-puter vision cannot achieve to a high degree of precision. However,human vision is remarkably tolerant of certain types of physicalinaccuracies. It is therefore not necessary to achieve physically ac-curate results in order to produce a visually satisfying illusion ofmaterial transformation.

We have developed a sequence of simple heuristics for modify-ing the appearance of materials. Surprisingly, each component ofthe system on its own can be a gross simplification—indeed prov-ably wrong—as long as theensemble effect of the manipulationsrecreates the characteristic ‘look’ of the desired material.

Our system is capable of transformations ranging from texturingthe surface of an object, modelling transparency and translucencyto the application of any arbitrary bi-directional reflectance distribu-tion function (BRDF). Examples of some of our results are shownin Figures 1 and 2. The utility of this work lies in the facilitation ofdigital content creation for art and design, as well as visual effects.

2 Approach

To transform an object’s appearance, we first derive an approximateestimate of the environment from the image. This environment isthen modified and mapped onto the surface using an approximateestimate of the object’s 3D surface normals. Even though both en-vironment and 3D shape are only very roughly estimated from theimage, the resulting visual impression is surprisingly compelling.

The appearance of the material depends on how the normals areused to access the environment and on how the environment is pro-cessed. For example, the normals can be used simply to map anarbitrary texture directly onto the surface [Fang and Hart 2004].The normals can also be used to evaluate BRDFs, which allows usto apply arbitrary reflectance properties to the surface. Finally, theycan be used to map patterns from the background image onto thesurface, as if refracted through the object. This mimics the effectsof transparency or translucency.

Creating an environment involves removing the object from theimage. Several sophisticated ‘object removal’ algorithms exist forfilling in the background. However, we have found that a crudebut fast hole-filling technique is sufficient for this application. This

Figure 1:A statue is made transparent using our techniques.

Figure 2:Given a high dynamic range image such as shown on theleft, we present techniques to make objects translucent, as well asapply arbitrary surface materials (right).

works because the resulting patterns are highly distorted when ‘re-fracted’ through the object and thus errors in the hole-filling processhave negligible visual consequences.

To create translucent effects, the environment is color-correctedand blurred before refraction. Even though this is physically incor-rect, the object acquires the smooth, glowing appearance that weassociate with translucent objects.

Reconstructing 3D shape from a single image is an unsolvedproblem. However, we have found that the nature of the texturesbeing mapped is such that the object’s shape does not need to bevery accurate. Surprisingly, treating image intensities as depth es-timates and smoothing them with a bilateral filter [Oh et al. 2001]can yield visually satisfying results, as shown in Figures 1 and 2.In summary, it is possible to recreate theimage statistics associatedwith a particular material even without getting the physics right. Wehave exploited this to achieve purely image-based material manip-ulations.

ReferencesFANG, H., AND HART, J. C. 2004. Textureshop: Texture synthesis as a photographic

editing tool.ACM Transactions on Graphics 23, 3, 354–358.

OH, B. M., CHEN, M., DORSEY, J.,AND DURAND, F. 2001. Image-based modelingand photo editing. InProceedings ACM SIGGRAPH 2001, 433–442.