A Review on Shadow Techniques in Augmented Reality

Zakiah Noh Department of Computer Graphics and Multimedia

Faculty of Computer Science and Information System Universiti Teknologi Malaysia 81310 Skudai, Johor, Malaysia

zakiahnoh@gmail.com

Mohd Shahrizal Sunar Department of Computer Graphics and Multimedia

Faculty of Computer Science and Information System Universiti Teknologi Malaysia 81310 Skudai, Johor, Malaysia

shah@fsksm.utm.my

Abstract— The seamless integration between computer-generated objects with real scene is desirable in Augmented Reality applications. The challenging task in Augmented Reality application is to make virtual objects mixed harmoniously with the real scene. To achieve these photorealistic rendering, it is not only considering visual complexity of virtual objects but the consistency of illumination between virtual objects with real scene must also be determined. With the consistency of illumination, virtual objects can cast shadows that appear as part of the real scene. The consistent shadow of the virtual objects will give a correct explanation of the relative distance between the virtual and the real objects. This paper will explore regarding shadow techniques to improve photorealism for rendering virtual objects in Augmented Reality system. The evaluation among previous techniques presented in this paper is also important as a preliminary step for future findings related to the field of Augmented Reality.

Keywords-Augmented Reality(AR); Illumination; Shadow

I. INTRODUCTION The seamless integration between real and virtual worlds

is still challenges topics among researchers in Augmented Reality (AR) applications. AR is a technology that combines real and virtual objects in the same space. This technology displays computer generated images in the real worlds and to achieve visual realism, they should appear as part of the real scene. However, in some AR applications, the visual realism is not exacting important for instance blending the visual objects in the concealed structure such as wires and cables inside walls or other solid elements [1].

To achieve photorealistic rendering in AR, three aspects have been identified to achieve a high quality of AR system [2]. There are geometry consistency, illumination consistency and time consistency. Consistency of geometry is referring to the correct position of virtual object in the real scene location. Meanwhile, the consistency of illumination is matching the shading of virtual object with other object in the real scene and the virtual object must cast a correct shadow. Lastly, synchronized the real world and virtual world to facilitate smooth interaction is a consistency of time.

In this paper, we want to focus on the consistency of illumination which highlight on shadow. The study of shadow techniques in AR system will be explorer to give the

user the overview of shadow applied in AR. Shadow is one of the elements that essential for the improvement of visual perception and realism in AR environment. The consistent shadow of the virtual objects will give a correct explanation of the relative distance between the virtual and the real objects.

The rest of the paper is organized as follows. In Section II, we briefly describe the AR technology. Section III will explain about shadow in AR system and timelines of shadow technique in AR system will describe in section IV. Finally, in Section V, we present the discussion and conclusion related with the topic presented.

II. AUGMENTED REALITY According to [3], in 1994, the context of AR, often the

term Mixed Reality (MR) is used. It is a superset of AR and covers the reality-virtuality continuum between completely real environments and completely virtual environments, a concept introduced by Milgram and Kishino, which encompasses both AR and Augmented Virtuality (AV), as shown in Figure 1 below.

Figure 1. Milgram’s Reality-Virtuality Continuum and examples of each

fields of reality

AR allows a user to work with real environments while

visually receiving additional computer generated or modeled information about the task at hand. Such displays can enhance the user’s perception of the real environment by showing information that the user cannot directly sense when unaided. AR systems share the three following properties: first is blending real and virtual in a real environment. Secondly is running in real-time and the third is registered in 3D [3].

2009 Second International Conference on Machine Vision

978-0-7695-3944-7 2010

U.S. Government Work Not Protected by U.S. Copyright

DOI 10.1109/ICMV.2009.41

320

In constructing AR system, display technology continues to be a limiting factor in the development of AR systems. There are several needs to accomplish the combining of real and virtual objects. Two basic choices are available: optical and video technologies [4] [5].

Optical see-through Head-Mounted Displays (HMDs) work by placing optical combiners in front of the user's eyes. These combiners are partially transmissive, so that the user can see the real world directly through them. In contrast, video see-through HMDs work by combining a closed-view HMD with one or two head-mounted video cameras. The video cameras provide the user's view of the real world. Video from these cameras is combined with the graphic images created by the scene generator, blending the real and virtual. The result is sent to monitors in front of the user's eyes in the closed-view HMD [5][6].

AR applications become feasible in multidisciplinary areas such as in simulation, education, entertainment, medical and game. Furthermore, a lot of effort had being done by researchers to improve the quality of AR system. Every aspects in AR system such as registration, collabaration, illumination and geometrical issues is analyze and improve to produce the higher quality of the system. Thus, the next section will discuss the illumination which focus on shadow. The shadow is signification to add the realism of the AR system.

III. SHADOW IN AUGMENTED REALITY The photorealistic AR environment can be enhanced with

adding shadow to the environment. This is because shadows can improve the image realism and add visual perception of the users. When at least one light source is occluding by the object, also known shadow caster, occlude or blocker, shadow will be appear [7]. Figure 2 illustrates the generation of shadow. The union of the umbra and penumbra is the shadow.

Figure 2. A shadow caster casts an umbra and penumbra region when illuminated by an area light source.

Shadows have two types: attach shadows and cast shadows. When the normal of the receiver is facing away from the light source, attach shadows will occurred. Meanwhile, cast shadows occurring when a shadow fall on an object whose normal is facing toward the light source.

Furthermore, cast shadows has specific shadow referring self-shadows. Self- shadows occur when the shadow of an object is projected onto itself [9].

Shadows are important to give more information of the size, position and shape of objects. This information is significant to improve objects presence. Without shadows, computer generated images will look unreasonable even with precise measurements of the light source and material properties [8]. Currently, many researches develop new techniques to create realistic shadow especially in augmented reality. The next section will discuss the timelines of shadow in AR system.

IV. TIMELINE OF SHADOW IN AUGMENTED REALITY With the increasing of development computer graphics

technologies, a lot of researches in this field have being done to improve the quality of the graphics system. In AR system, the research about shadow generation is growth rapidly. This shows that, the shadows became one of the most important elements in AR environment to achieve photorealistic rendering. Figure 3 shows the timeline of shadow techniques.

Figure 3. Timelines of shadows techniques

A radiance giving out within shadows emit by an object in the scene is used as a method by Sato et al. [10] for approximation the illumination distribution of a real scene by the occlusion information of the arriving light source. Then, the virtual objects are rendering with shading and shadowing to conform to that environment. Their method is useful for approximate an illumination giving out even in a complex illumination environment. Results from this method are used to produce photorealistic virtual objects with illumination consistency. Kim et al. [11] extend the work by Sato with proposed a method for generating graphic objects having realistic shadows inserted into video sequence using virtual camera for enhance AR. This proposed method improved the limitation of a static camera in [10].

To attain a mixed reality environment that contains a shadow, Naemura et al. [12] proposed the concepts of virtual light and virtual shadow. The concept of virtual shadow in this method is dividing into four types: i) real to virtual shadow for rigid objects, ii) real to virtual shadow for non-rigid objects, iii) image-based virtual to virtual shadow, and iv) virtual to real shadow. These methods will project the shadow of a real object onto the virtual world and vice versa.

321

A natural merge between real and virtual worlds will be get when the shading and shadows correspond between these two worlds.

Figure 4. The entire scene is drawn based on the content of the stencil

buffer with shadows.

The concept for generate real-time shadow for AR applications using shadow volume was proposed by Haller et al. [13]. This proposed methods more focused on the shadow problems in AR system. The reality of AR worlds improved with projected real shadows onto virtual objects and vice versa. Figure 4 shows the result of the proposed method. Simultaneously, Sugano et al. [8] and Madsen et al. [1] highlight the importance of consistent shadows in AR environment. Madsen develop the method which automatically approximate the dominant characteristics of the lighting conditions in the real world, and uses these characteristics when rendering the virtual objects. These techniques make possible to get quite persuasive results in real-time. Figure 5 illustrated the result of their work. The lighting of the virtual objects as well as the virtual shadows is geometrically and spectrally consistent with the lighting in the real scene which important to greatly improved realism. Meanwhile, Sugano express the effects of shadow depiction by virtual objects in AR environment from the viewpoint of illumination consistency. The technique used in this work is shadow mapping which is a very accepted and efficient technique for virtual imagery in real-time rendering.

Gibson et al. [14] presented a shadow rendering algorithm appropriate for photorealistic AR environment. Their algorithm combines a shaft-based hierarchical data structure with a technique that allows soft shadows to be estimated using multiple shadow-maps. These combinations were mixed into the background image using commonly available computer graphics hardware. Shaft-based hierarchical data structure is used to allow fast identification of the sources of light that are occluded by an artificial object. Virtual objects emit the shadows and combined into a background image quick sufficient to allow interactive object manipulation.

Figure 5. Some virtual objects have been added to the scene taking occlusions into account with shadows cast be virtual objects added.

A real-time rendering solution to simulate color-consistent of virtual shadows in a real environment was present by Jacobs et al. [15]. The rendering process in their proposed method has three step mechanisms; shadow detection, shadow protection and shadow generation which every step produces consistent shadows between real and virtual objects in real-time. To accomplish the successful shadow detection and shadow generation in their method, three requirements are need: the geometry and the light source position require to be known approximately and only hard or semi-soft shadows are allowed. The results of their proposed solution are shown in Figure 6.

Figure 6. Results of the algorithm for an animated virtual avatar walking

around a real laptop. Shadows are automatically detected and generated using real-time algorithm.

Kakuta et. al [16] extend Sato et al.’s method for interactive mixed reality application. They create basis images only to express the shadow of virtual objects and set them on other shadowing planes in order that they correspond to both the random viewpoints and altering illumination of the real environment. However, the proposed method can be relevant in outdoor scene at static models such as architecture and can yield result in near real-time.

Image-based shadowing technique proposed by Supan et al. [17] was used to create soft shadows by calculating shadow buffer from downsampled light source map. To attain photometric consistency in AR system, their research demonstrates the necessity of reducing the resolution of sensing and rendering. Furthermore, to achieve interactive frame rates, the system based on both vertex and pixel shaders.

Madsen and Laursen [18] present a complete AR rendering system using High Dynamic Range environment

322

maps for representing the real world illumination. Their approach capable for shading virtual object and render shadows in AR system which is consistent with the illumination in the real scene as depicted in Figure 7. The system allows for scaling the visual quality with the increasing performance of graphics hardware in the sense that as hardware improves more and more light sources can be included in the set of sources which are allowed to cast shadows.

Figure 7. Screenshots demonstrating ability to mix multiple shadows and

handle the double shadow problem.

Nakano et al. [19] discover the resolution of a light source map to produce perceptually-correct shadow. The information of this resolution was got from the performed a set of logical subjective assessment to get information about the relation between the size and arrangement of light sources and the human perception of the shadows produced by light sources.

The method for estimate the illumination conditions in outdoor environments from a single image was proposed by Madsen [20]. This proposed method is not require acquisition of a light probe image which assume the some additional information such as date and time, position for longitude and latitude of earth where image is taken and direction of gravity relative to camera are available for the image acquisition. This information is straight forward to compute the position of the Sun relative to the camera coordinate system. The proposed method based on shadows present in the image which can approximation the radiances from the Sun and the Sky. Using these radiances, augmented objects will be render into the images with very realistic illumination regarding both shading and shadowing. Figure 8 shows the result of developed technique which renders virtual objects into an image.

Figure 8. Image with virtual objects rendered with estimated illumination.

V. CONCLUSION AND DISCUSSION Shadows in AR become one of the most important

elements to improve the photorealistic rendering in AR environment. It gives more information such as size, position and shape of objects. This information is important to enhance objects presence. This paper focuses on overview of shadows approach in AR system. The timelines of the shadows generation have been discussed including current approach of research in AR technology. Various methods were developed by the researches in the effort to improve the quality of AR environment and increase the realism of user viewpoint especially in shadow generation.

As a conclusion, this study will be useful for the AR system developer and therefore, contributing to the computer graphics community. Nonetheless, this will bring benefits especially for academic purposes and in research fields related to AR system.

ACKNOWLEDGMENT We would like to express our appreciation to Universiti

Teknologi Malaysia (UTM) and Malaysian Ministry of Science, Technology and Innovation (MOSTI) for providing financial support for this research through eScienceFund grant.

REFERENCES [1] C.B. Madsen, M.K.D. Sorensen, M. Vittrup, “The important of

shadows in Augmented Reality,” In Prpceedings 6th Annual International Workshop on Presence, Aalborg, Denmark, 2003

[2] Sato I, Sato Y and Ikeuchi K, “Acquiring a radiance distribution to superimpose virtual objects onto a real scene,” IEEE Transactions on Visualization and Computer Graphics, 5(1), 1999, pp.1–12.

[3] P. Milgram, F. Kishino, “A Taxonomy of Mixed Reality Visual Displays,” IEICE Trans. Information Systems, 1994, pp. 1321-1329.

[4] R. Azuma, “A Survey of Augmented Reality,” Presence: Teleoperators and Virtual Environments, 1997, pp. 355-385.

[5] R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier and M. Blair, “Recent advances in Augmented Reality,” IEEE Computer Graphics and Applications, 2001, pp. 20-38.

[6] A.W. Ismail and Z. Noh, “Advances in Computer Graphic & Virtual Environment”, M.S. Sunar and M.N. Zamri (Eds), Universiti Teknologi Malaysia, 2008, pp. 87 – 105.

[7] D. Scherzer, “Shadow Mapping of Large Environments,” Vienna University of Technology, Master’s Thesis, 2006.

[8] N. Sugano, H. Kato and K.Tachibana, “The effects of shadow representation of virtual objects in Augmented Reality,” In IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR 2003), IEEE Computer Society, 2003, pp. 76-83.

[9] J. Hasenfratz, M. Lapierre, N. Holzschuch, and F. Sillion, “ A survey of real-time soft shadows algorithms,” In Eurographics, 2003.

[10] I. Sato, Y.Sato and K. Ikeuchi, “Illumination distribution from brightness in shadows: adaptive estimation of illumination distribution with unknown reflectance properties in shadow regions,” In Proc. Int. Conf. on Computer Vision, 1999, pp 875-882.

[11] T. Kim, K.S. Hong, I. Sato, and K. Ikeuchi, “Enhanced Augmented Reality with shadow in naturally illuminated environmnent,” IAPR Workshop on Machine Vision Applications, The University of Tokyo, Japan, 2000.

[12] T.Naemura, T. Nitta, A. Mimura, and H. Harashima, “Virtual shadows in mixed reality environment using flashlight-like devices,” Trans. Virtual Reality Society of Japan, vol. 7(2), 2002, pp. 227-237.

323

[13] M. Haller, S. Drab, and W. Hartmann, “A real-time shadow approach for an augmented reality application using shadow volumes,” In Proceedings of VRST 03, 2003, pp. 56–65.

[14] S. Gibson, J. Cook, T. Howard, and R. Hubbold, “Rapid shadow generation in real-world lighting environments,” In Proceedings of Eurographics Symposium on Rendering 03, 2003, pp. 219–229.

[15] K. Jacobs, C. Angus, and C. Loscos, “Automatic generation of consistent shadows for augmented reality,” In Proceedings: Graphics Interface, Vancouver, Canada, 2005.

[16] T. Kakuta, T. Oishi, and K. Ikeuchi, “Shading and Shadowing of architecture in mixed reality,” In Proceedings of ISMAR 2005, 2005.

[17] P. Supan, I. Stuppacher and M. Haller, “Image Based Shadowing in Real-Time Augmented Reality,” in International Journal of Virtual Reality, vol. 5, issue. 3, 2006, pp 1-7.

[18] C. B. Madsen, and R. Laursen, “A scalable GPU-based approach to shading and shadowing for photo-realistic real-time augmented reality,” In Proceedings: International Conference on Graphics Theory and Applications, Barcelona, Spain, 2007, pp. 252 – 261.

[19] G. Nakano, I. Kitahara and Y. Ohta, "Generating perceptually-correct shadows for mixed reality," 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, 2008, pp.173-174.

[20] C. B. Madsen, “Estimating Radiances of the Sun and the Sky From a Single Image Containing Shadows,” Proceedings: 16th Danish Conference on Pattern Recognition and Image Analysis. Department of Computer Science. University of Copenhagen, 2008.

[21] I. Sato, Y. Sato and K. Ikeuchi, "Illumination from Shadows," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 3, pp. 290-300, 2003.

[22] T. Grosch, T. Eble and S. Mueller, “Consistent interactive augmentation of live camera images with correct near-field illumination,” Proceedings of the 2007 ACM symposium on Virtual reality software and technology, Newport Beach, California, 2007, pp. 125 – 132.

[23] T. Kakuta, L. B. Vinh, R. Kawakami, T. Oishi, K. Ikeuchi, "Detection of Moving Objects and Cast Shadows Using a Spherical Vision Camera for Outdoor Mixed Reality," 15th ACM Symposium on Virtual Reality Software and Technology(VRST 2008), 2008.

324