Image-Based Material Editing - Erik Reinharderikreinhard.com › papers › sketch.pdf · In...
1
Image-Based Material Editing Erum Khan Erik Reinhard University of Central Florida Roland Fleming Heinrich B¨ ulthoff Max Planck Institute for Biological Cybernetics Abstract Photo editing software allows digital images to be blurred, warped or re-colored. However, it is not currently possible to change the material appearance of an object except by painstakingly painting over the appropriate pixels. We present a set of methods for au- tomatically replacing one material with another material, starting with only a single high dynamic range (HDR) image as input. 1 Objectives We address the problem of editing a photograph of an object—say a bronze sculpture or a porcelain vase—so that it appears to be made out of some completely different material, such as glass, plastic, or wax. To do this in a physically correct way would involve accu- rately estimating the lighting, material properties and 3D geometry of the object from the image; a task that even state-of-the-art com- puter vision cannot achieve to a high degree of precision. However, human vision is remarkably tolerant of certain types of physical inaccuracies. It is therefore not necessary to achieve physically ac- curate results in order to produce a visually satisfying illusion of material transformation. We have developed a sequence of simple heuristics for modify- ing the appearance of materials. Surprisingly, each component of the system on its own can be a gross simplification—indeed prov- ably wrong—as long as the ensemble effect of the manipulations recreates the characteristic ‘look’ of the desired material. Our system is capable of transformations ranging from texturing the surface of an object, modelling transparency and translucency to the application of any arbitrary bi-directional reflectance distribu- tion function (BRDF). Examples of some of our results are shown in Figures 1 and 2. The utility of this work lies in the facilitation of digital content creation for art and design, as well as visual effects. 2 Approach To transform an object’s appearance, we first derive an approximate estimate of the environment from the image. This environment is then modified and mapped onto the surface using an approximate estimate of the object’s 3D surface normals. Even though both en- vironment and 3D shape are only very roughly estimated from the image, the resulting visual impression is surprisingly compelling. The appearance of the material depends on how the normals are used to access the environment and on how the environment is pro- cessed. For example, the normals can be used simply to map an arbitrary texture directly onto the surface [Fang and Hart 2004]. The normals can also be used to evaluate BRDFs, which allows us to apply arbitrary reflectance properties to the surface. Finally, they can be used to map patterns from the background image onto the surface, as if refracted through the object. This mimics the effects of transparency or translucency. Creating an environment involves removing the object from the image. Several sophisticated ‘object removal’ algorithms exist for filling in the background. However, we have found that a crude but fast hole-filling technique is sufficient for this application. This Figure 1: A statue is made transparent using our techniques. Figure 2: Given a high dynamic range image such as shown on the left, we present techniques to make objects translucent, as well as apply arbitrary surface materials (right). works because the resulting patterns are highly distorted when ‘re- fracted’ through the object and thus errors in the hole-filling process have negligible visual consequences. To create translucent effects, the environment is color-corrected and blurred before refraction. Even though this is physically incor- rect, the object acquires the smooth, glowing appearance that we associate with translucent objects. Reconstructing 3D shape from a single image is an unsolved problem. However, we have found that the nature of the textures being mapped is such that the object’s shape does not need to be very accurate. Surprisingly, treating image intensities as depth es- timates and smoothing them with a bilateral filter [Oh et al. 2001] can yield visually satisfying results, as shown in Figures 1 and 2. In summary, it is possible to recreate the image statistics associated with a particular material even without getting the physics right. We have exploited this to achieve purely image-based material manip- ulations. References FANG, H., AND HART, J. C. 2004. Textureshop: Texture synthesis as a photographic editing tool. ACM Transactions on Graphics 23, 3, 354–358. OH, B. M., CHEN, M., DORSEY, J., AND DURAND, F. 2001. Image-based modeling and photo editing. In Proceedings ACM SIGGRAPH 2001, 433–442.
Transcript of Image-Based Material Editing - Erik Reinharderikreinhard.com › papers › sketch.pdf · In...
Image-Based Material Editing
Erum Khan Erik ReinhardUniversity of Central Florida
Roland Fleming Heinrich B̈ulthoffMax Planck Institute for Biological Cybernetics
Abstract
Photo editing software allows digital images to be blurred, warpedor re-colored. However, it is not currently possible to change thematerial appearance of an object except by painstakingly paintingover the appropriate pixels. We present a set of methods for au-tomatically replacing one material with another material, startingwith only a single high dynamic range (HDR) image as input.
1 Objectives
We address the problem of editing a photograph of an object—say abronze sculpture or a porcelain vase—so that it appears to be madeout of some completely different material, such as glass, plastic, orwax. To do this in a physically correct way would involve accu-rately estimating the lighting, material properties and 3D geometryof the object from the image; a task that even state-of-the-art com-puter vision cannot achieve to a high degree of precision. However,human vision is remarkably tolerant of certain types of physicalinaccuracies. It is therefore not necessary to achieve physically ac-curate results in order to produce a visually satisfying illusion ofmaterial transformation.
We have developed a sequence of simple heuristics for modify-ing the appearance of materials. Surprisingly, each component ofthe system on its own can be a gross simplification—indeed prov-ably wrong—as long as theensemble effect of the manipulationsrecreates the characteristic ‘look’ of the desired material.
Our system is capable of transformations ranging from texturingthe surface of an object, modelling transparency and translucencyto the application of any arbitrary bi-directional reflectance distribu-tion function (BRDF). Examples of some of our results are shownin Figures 1 and 2. The utility of this work lies in the facilitation ofdigital content creation for art and design, as well as visual effects.
2 Approach
To transform an object’s appearance, we first derive an approximateestimate of the environment from the image. This environment isthen modified and mapped onto the surface using an approximateestimate of the object’s 3D surface normals. Even though both en-vironment and 3D shape are only very roughly estimated from theimage, the resulting visual impression is surprisingly compelling.
The appearance of the material depends on how the normals areused to access the environment and on how the environment is pro-cessed. For example, the normals can be used simply to map anarbitrary texture directly onto the surface [Fang and Hart 2004].The normals can also be used to evaluate BRDFs, which allows usto apply arbitrary reflectance properties to the surface. Finally, theycan be used to map patterns from the background image onto thesurface, as if refracted through the object. This mimics the effectsof transparency or translucency.
Creating an environment involves removing the object from theimage. Several sophisticated ‘object removal’ algorithms exist forfilling in the background. However, we have found that a crudebut fast hole-filling technique is sufficient for this application. This
Figure 1:A statue is made transparent using our techniques.
Figure 2:Given a high dynamic range image such as shown on theleft, we present techniques to make objects translucent, as well asapply arbitrary surface materials (right).
works because the resulting patterns are highly distorted when ‘re-fracted’ through the object and thus errors in the hole-filling processhave negligible visual consequences.
To create translucent effects, the environment is color-correctedand blurred before refraction. Even though this is physically incor-rect, the object acquires the smooth, glowing appearance that weassociate with translucent objects.
Reconstructing 3D shape from a single image is an unsolvedproblem. However, we have found that the nature of the texturesbeing mapped is such that the object’s shape does not need to bevery accurate. Surprisingly, treating image intensities as depth es-timates and smoothing them with a bilateral filter [Oh et al. 2001]can yield visually satisfying results, as shown in Figures 1 and 2.In summary, it is possible to recreate theimage statistics associatedwith a particular material even without getting the physics right. Wehave exploited this to achieve purely image-based material manip-ulations.
ReferencesFANG, H., AND HART, J. C. 2004. Textureshop: Texture synthesis as a photographic
editing tool.ACM Transactions on Graphics 23, 3, 354–358.
OH, B. M., CHEN, M., DORSEY, J.,AND DURAND, F. 2001. Image-based modelingand photo editing. InProceedings ACM SIGGRAPH 2001, 433–442.
