Multimed Tools ApplDOI 10.1007/s11042-013-1630-6

A quadratic spline approximation using detailmulti-layer for soft shadow generationin augmented reality

Hoshang Kolivand · Zakiah Noh ·Mohd Shahrizal Sunar

© Springer Science+Business Media New York 2013

Abstract Implementation of shadows is crucial to enhancement of images in ARenvironments. Without shadows, virtual objects would look floating over the sceneresulting in unrealistic rendering of AR environments. Casting hard shadows wouldprovide only spatial information while soft shadows help improve realism of ARenvironments. Several algorithms have been proposed to render realistic shadowswhich often incurred high computational costs. Little attention has been directedtowards the balanced trade-off between shadow quality and computational costs.In this study, two approaches are proposed: Quadratic Spline Interpolation (QSI)to soften the outline of the shadow and Detail Multi-Layer (DML) technique tooptimize the volume of computations for the generation of soft shadows basedon real light sources. QSI estimates boarder hard shadow samples while DMLinvolves three main phases: real light sources estimation, soft shadow production andreduction of the complexity of 3-Dimensional objects’ shadows. To be more precise, areflective hemisphere is used to capture real light and to create an environment map.The Median Cut algorithm is implemented to locate the direction of real light sourceson the environment map. Subsequently, the original hard shadows are retrieved anda sample of multilayer hard shadows is produced where each layer has its uniquesize and colour. These layers overlap to produce soft shadows based on the real lightsources’ directions. Finally, the Level of Details (LOD) algorithm is implementedto increase the efficiency of soft shadows by decreasing the complexity of vertextransformations. The proposed technique is tested using three samples of multilayerhard shadows with varying numbers of light sources generated from the Median Cutalgorithm. The experimental results show that the proposed technique successfullyproduces realistic soft shadows at low computational costs.

H. Kolivand (B) · Z. Noh · M. S. SunarUTM ViCubelab, Department of Computer Graphics and Multimedia,Faculty of Computer Science and Information Systems,Universiti Teknologi Malaysia, 81310 Skudai Johor, Malaysiae-mail: shahinkey@yahoo.com

Multimed Tools Appl

Keywords Augmented reality · Shadow generation · Soft shadows ·Reflective sphere · Environment map

1 Introduction

It hardly needs stressing that Augmented Reality (AR) has the potential to beapplied as a fascinating and widespread technology not only in computer graphicsbut also in many other subjects. Over the last two decades, Mixed Reality (MR) hasbeen one of the most attractive topics in computer graphics motivating researchersto achieve better results in this field [4, 17, 19, 31]. Through shadows and realillumination on virtual objects, MR is able to create reality [1]. Madsen et al. [32]and Sugano et al. [38] have incorporated the ability of consistent shadows in ARenvironments to create realistic effects.

Mixed Reality is the integration of Virtual Environments (VE) and Real Environ-ments (RE). A virtual object set within a real environment constitutes an AugmentedReality system. An AR-system incorporates more real objects and a few virtualobjects with the real AR taking a dominant role over the virtual. On the other hand,if a real object is set within a virtual environment, the system is called Virtual Reality(VR). In this case, most of the system is considered as being virtual. In general,MR can be characterized by the integration of virtual and real objects, real-timeinteraction and 3D registration. In addition to including some virtual objects withinthe real environment, augmented reality makes it possible to remove or hide someobjects in real environments [5, 35].

AR technology is used in various fields such as medicine, education, manufactur-ing, advertising, gaming and tourism. AR technology is applied in certain importantareas including learning, training, entertainment, information resources and explo-ration. It is also used in planning strategies and solving problems in which precisionis required, and high cost and risks are inevitable. In this era of globalization, ARtechnology plays a significant role in the development of information technologyboth in the present time and in future.

To achieve photorealism in AR systems, the integration between virtual objectsand real environments must be corrected. According to Sato et al. [36], three princi-pal aspects are identified to achieve high quality AR systems including consistencyin terms of geometry, illumination and time. Consistency in geometry refers tocorrection of locations for virtual objects in real environments, and consistency interms of illumination is concerned with similarities between lighting of virtual objectswith other objects in real environments while creating virtual shadows similar toreal shadows [29]. Consistency in terms of time deals with the simultaneous andparallel movement of virtual objects with real objects in mixed environments so thatinteractions and the resulting motions appear smooth and realistic [24, 28, 37].

This research addresses the impact of illumination leading to the generation ofshadows in AR environments. Shadows are one of the most important aspects inmixed reality environments. Shadows make it possible for the spectator to detectdistance relationships between objects making the scene more realistic. Virtualobjects lacking shadows in real environments result in unrealistic AR environmentsespecially when real objects have shadows. Generally, virtual objects appear likefloating things in AR environments. However, some shadow generation techniquesproduce soft shadows although at a high computational cost. This is due to the high

Multimed Tools Appl

usage of hardware resources at that time. Therefore, to consider a balanced trade-off between shadow quality and computational cost, a Detail Multi-Layer (DML)technique is developed. The approach is based on a Quadratic Spline Interpolation(QSI) to soften the outline of the shadow and DML technique to optimize the volumeof computations for the generation of soft shadows based on real light source.

The rest of the paper is organized as follows: Section 2 gives brief explanationsof related works on shadows in AR. Section 3 discusses the process of shadowimplementation in AR environments and which is continued with experiments inSection 4. Experimental results are discussed in Section 5. Section 6 comparesthe results obtained from state-of-the-art in shadow generation in AR and finally,Section 7 concludes this paper with recommendations for future works.

2 Related works

Research on shadows in AR environments was initiated near two decades ago.Among researchers who conducted research on shadows in AR environments onecan refer to Naemura et al. [33], Sugano et al. [38], Gibson et al. [10], Haller et al.[11, 12], Madsen et al. [32], Kanbara and Yokoya [22], Hughes et al. [18], Jacobs et al.[19], Madsen and Laursen [30], Jensen et al. [20] and Kolivand et al. [24]. Consideringthe problems and deficiencies found in current research such as the trade-off betweenrendering speed and realism, it is necessary to continue studies to improve shadowsin AR environments whereby the problems can be reduced taking the trade-off intoaccount.

Naemura et al. [33] proposed the concept of virtual light and virtual shadows. Theconcept of virtual shadows in this method is divided into four types including real tovirtual shadows for rigid objects, real to virtual shadow for non-rigid objects, image-based virtual to virtual shadows, and virtual to real shadows. These methods projectshadows of real objects onto virtual world and vice versa. A natural merge betweenreal and virtual worlds will be obtained where shading and shadows correspond inthe two worlds. Sugano et al. [38] and Madsen et al. [32] have also highlighted theimportance of consistent shadows in AR environments.

With regard to shadow performance, some researchers conducted in [9, 10, 30], inwhich the proposed approaches are designed to run in graphics hardware offeringan approach to balance the performance without sacrificing the visual quality ofshadows. Besides that, generating shadows using shadow maps [25, 30, 38] or shadowvolumes [12, 23] can be developed at low costs. Meanwhile, Haller et al. [12]proposed the use of shadow volumes which is focused more on the shadow problemsin AR systems. The reality of AR world is improved with real shadows projectedonto virtual objects and vice versa.

A survey of soft shadow classification method was done in [14]. These methodscan be applied to generate shadows in the context of AR environments. Jacobset al. [19] and Madsen and Laursen [30] pointed out the issues of double shadowin AR environments where they had solved the problem of overlap between realand virtual shadows to produce realistic shadows. A real-time solution to simulatecolour-consistent virtual shadows in real environments was presented by Jacobs et al.[19]. The rendering process in their proposed method has a three-step mechanismcomprising of shadow detection, shadow protection and shadow generation. Every

Multimed Tools Appl

step produces consistent shadows between real and virtual objects in real-time. Toaccomplish successful shadow detection and shadow generation in their method, it isnecessary to fulfill three requirements. In fact, Geometry, light source position andonly hard or semi-soft shadows should be taken into consideration.

The credibility of shadow creation can also be achieved through estimation of lightsource position. Research related to light source position can be found in [20, 22, 39].All the studies have the same objective which is to propose a method capable ofcreating lighting for virtual objects in AR environments as realistically as possiblelike in the real world. The present research is based on principles and lessons fromthese research undertakings, which has produced soft shadows with regard to reallight source direction.

The current work is constructed based on projection shadows. Some improve-ments are employed to enhance the quality of soft shadows. Regarding projectionshadows, the rendering time compared to current related works is needed. Manyresearchers have attempted to improve the soft shadow quality in AR but notmuch attention has been given to rendering time. Nakano et al. [34] introduced atechnique to find the resolution of the light sources maps to create perceptually-correct shadows. Yeoh et al. [40] proposed a technique for realistic shadows in ARusing a shadow segmentation approach which recovered geometrical informationon multiple faded shadows. The paper focused on dynamic shadow detection ina dynamic scene for future requirements in augmented reality environments. Thetechnique is similar to Shadow Catcher in Hartmann et al. [13] but in dynamicscenes. Aittala [2] applied Convolution Shadow Maps (CoSMs) [3] to produce softshadow in AR employing both mip-map filtering and fast summed area tables [15] toenhance blurring with variable radius. Castro et al. [6] proposed a method to producesoft shadows with less aliasing using a fixed distance relative to the marker. Themethod also performs sphere mapping e.g. [22], but selects a source or sources oflight most representative of the scene. This is important due to hardware limitationsof mobile devices. The main problems of this method are related to sampling andaliasing depths. These problems are closely connected and depend on the resolutionfor calculating visibility.

We have proposed a technique to overcome the current issues in AR systemssuch as quality of soft shadows and rendering time. The technique enhances thequality of soft shadows in terms of the amount of penumbra and reduces renderingtime as it employs the Detail Multi-Layer (DML) technique to optimize the volumeof computations.

3 Methods and materials

The detailed process of generating realistic shadows is explained in this section. Aframework is created where the development process of shadows and the method ofsoft shadow creation is demonstrated.

3.1 Framework

The framework used in this research is shown in Fig. 1. The problems of consistencyof geometric and photometric registration as described in [21] are highlighted in

Multimed Tools Appl

Fig. 1 The QSI approximationfor a part shadow boundary

the implementation processes. These problems need to be resolved especially whendealing with unrealistic fake shadow generation. Then, shadows will be renderedbased on estimation of light source position in the real scene and correct-perceptionof users’ viewpoint will be obtained. To enhance the softness of shadow samples,QSI is employed, besides the level of details (LOD) of shadows and the dynamiclight sources reduce computational costs of shadow generation.

The main contribution of this research is to apply QSI to enhance the realism andto employ DML to reduce the rendering time for AR systems. QSI tries to convertthe sharp outline of hard shadow samples to a smooth outline, while DML optimizesthe number of layers, resulting in a reduction of rendering time.

3.2 Implementation

The system set-up in this research is illustrated in Fig. 2. The set-up entails certaininstruments such as computer, camera, marker and light source. The marker isconstructed using a reflective sphere and a 2D marker. The camera detects themarker with a reflective sphere in the real scene. The relationship between the 2Dmarker and the camera coordinate system is determined using an existing technique[7]. This relationship is a significant step to complete the geometric registrationso that the virtual object can be placed in the right position. The system detectsthe reflective sphere on the marker painted with glossy black paint with which the

Multimed Tools Appl

Fig. 2 System setup

dynamic range problem is avoided to create the environment map. The environmentmap is used to define the incoming light from all possible directions at some referencepoint using the median cut algorithm.

This algorithm produces a set of light sources [7] and estimates the light sourceposition in the real scene from the environment map. The light sources are used togenerate realistic shadows. The soft shadows using QSI is applied to obtain smooth-edge outlines of shadows so that the shadows look realistic. Finally, the 3D virtualobject with the correct attached shadow is rendered in the real scene.

3.3 Light source tracking

Reflective sphere is a spherical object with glossy surface applied to capture lightfrom surrounding environment. In this paper, the reflective sphere is frequently usedto obtain the position of the real light source. The purpose is to generate a virtualshadow with the same direction as the shadows in the real environment. The areaof the reflective sphere which appears on camera view needs to be segmented intothe image for the calculation purposes. Next, the image is segmented to the pixelcoordinates (Fig. 2) [22].

The reflection of light from the reflective sphere falls on the surface of the ball andbounces the light back into the surrounding environment and the camera. This meansthat each pixel in the segmentation of the circle represents the light reflected from thesurface of the reflective sphere. In this research, both the spherical coordinates andCartesian coordinates are used in mapping the reflective sphere. The image of thecircle segment is implemented using the forward mapping method. Forward mappingis implemented through each pixel in the circle segmentation of the reflective spherewhere every appropriate pixel value is mapped to the environment map [7, 20].

The Median Cut algorithm which is used in this research aims to locate the sourceof light in the light probe image. This algorithm converts the light probe images to agroup of light sources. The median cut algorithm splits the environment map imageinto 2n regions based on latitude and longitude format [7]. The regions which arealready split have an equal light energy stemming from a light source. The steps tosplit the environment map image are enumerated as follows.

1. The environment map of light probe is added to the region list as a single region2. Every region in the list will subdivide along the longest dimension so that its light

energy is split evenly

Multimed Tools Appl

3. Return to step 2 if the number of iterations is less than n4. The light source is placed at the center of each region, and the colour of the light

source is set to the sum of pixel values within the region

The advantages of using this algorithm include its being efficient, fast and easy toput into practice. The algorithm is immensely practical to be applied in merging thevirtual object into the real scene.

3.4 Soft shadows generation

Projection shadows are a fast rendering technique to generate shadows on flatsurfaces. In this method, the entire object is simply rendered a second time fromthe light source position. It is fast as it projects a shape on a flat receiver. There is noneed to use any buffers or create volume or detect the silhouette.

Soft shadow generation is still suffering from the trade-off between the quality andrendering time. An improvement is required to enhance the quality of soft shadows,while a strategy is needed to reduce the rendering time to maintain the trade-off.

The soft shadow method in this research is based on the concept of Herf andHeckbert [16] optimized with a new idea enhancing both sides of the trade-offbetween qualities and rendering speed. Our method produces a number of hardshadow samples. Each sample is a projection of occluder with respect to the lightsource onto the plane. Some hard shadow samples can be observed in Fig. 8b andc. These samples consist of shadows of different dark colours and sizes, where thesizes are slightly smaller and bigger than those of the original shadows. During theprocedure of reconstructing samples, the silhouette of samples is approximated byQSI. This approximation progressively renders the samples smoother. The numberof samples influences the quality of soft shadows. Figure 7a is a scene with 8 hardshadow samples for the bunny, while Fig. 7b is the same scene with 128 hard shadowsamples. In fact, the higher the number of samples used, the higher the quality of softshadows generated.

For a pixel (x, y) on the boundary of an original hard shadow sample with aconstant x:

y̆ = y + ε (1)

Where, (x, y̆) is a pixel on the plane on the silhouette of the improved hardshadow sample. ε represents the error for the original y which is computed bynormal distribution.

ε = N(0, 1) (2)

In this technique, the center of shadow area is recognized with respect to the lightsource by drawing a ray of light to the center of occluder. Then, the shadow outlineis assessed using the projection shadow technique by projecting the occluder on theplane [27]. By spreading away many pixels around the shadow outline with respectto the center of the shadow and applying QSI, a new approximation of shadowis constructed where the straight parts of shadow outline are changed with theapproximated interpolation as has been shown in Fig. 3. The technique is repeatedfor each sample to achieve high quality shadows. The technique has balanced thetrade-off between quality and rendering time.

Multimed Tools Appl

Fig. 3 Different approximation using different degrees of interpolation

Figure 3 illustrates how approximation can eliminate some straight shadow sil-houette with different degrees of spline interpolations. The QSI is selected due to itscapability of balancing the trade-off, otherwise most of the time high degrees wouldbe better to be used in finding the enhancement quality.

Having approximated all the new pixels with quadratic spline and extracted theshadow borders to the new approximation, a new smooth border is obtained underthe following assumption:

y̆i(x) = aix2 + bix + ci, x ∈ [xi, xi+1] (3)

Then

y̆ =∑

i

di+1 − di

2(xi + 1 − xi)(xi+1 − xi)

2 + di(xi+1 − xi) + yi (4)

Multimed Tools Appl

Fig. 4 The QSI approximationfor a part shadow boundary

Where

di = yi + 1 − yi

xi + 1 − xi− di−1, d0 = 0 (5)

For each (x, y) which in the shadow outline y̆ is the approximation of y to make theoutline smooth, as can be seen in Fig. 3. QSI changes the outline of shadow samplesto become softer. The QSI is more effective when the objects are complicated. Ascan be seen from Fig. 4, the sharp outline of shadow samples is changed to a softoutline. This technique enhances the quality of soft shadows.

Having produced the few hard-shadow samples, the samples are blended togetherin different dark colours and sizes. This process is performed by stacking on each

Fig. 5 a Three level of details, b the process of QSI approximation to achieve the soft shadow witha few samples

Multimed Tools Appl

other starting from the lighter dark colour and ending with the original colour of theshadow. Figure 5a shows the concept of level of details in three levels, while Fig. 5billustrates the concept of soft shadow creation using the length and gap factors.

In producing soft shadows, this technique takes advantage of two parameterscalled the length and gap factors. The length factor determines the lengths of softshadows from the original hard shadows, while the gap factor determines the distancebetween hard shadows in the samples.

In summary, the light source position is recognized by Median Cut algorithm,which can be eliminated if the light source is controlled manually. Many hard samplesare constructed based on changing the location of light source. QSI covers the qualityof the object while keeping high FPS which is due to using the low level of LOD togenerate shadows especially during the movement of objects.

3.5 Implementation technique for level of detail of soft shadows

The development of soft shadows based on real light source involves a high com-putational cost. This is because of the fact that soft shadows having several layersare produced based on the use of a number of light sources which can be seen inFig. 6a. QSI is applied to enhance the quality while DML is employed to reducethe computational cost of soft shadows. Levels of detail are done on soft shadowsof virtual objects based on the visibility of the shadows with the camera position. Inthis research, two categories of details have been used. The first category is relatedto performing the LOD on the geometry of soft shadows and the second category isconcerned with doing dynamic light sources based on the LOD concept as illustratedin Fig. 6b.

The level of details for soft shadows used in this research involves three layers.Each level is represented by a virtual object which has a different number of vertices.For the first level, the number of used triangles equals 69,666. The number oftriangles, for the second level and the third level are 12,673 and 756, respectively(Fig. 5a). The main object used to represent the shadow is an object which has 69,666triangles. QSI is applied on each level of LOD and improves the soft shadow outputresulting in high quality of the original object with high numbers of triangles. QSI

Fig. 6 a Estimating light sources’ directions, b LOD illustration of soft shadows

Multimed Tools Appl

covers the quality of the object while keeping high FPS which is due to using the lowlevel of LOD to generate shadows especially during the movement of objects.

In general, this virtual object performs the movement. In the process of reach-ing a predetermined certain distance, details of the shadows for the virtual ob-ject are exchanged. In this research, only the details of the shadows are takeninto consideration, while the details of virtual objects producing the shadows arenot considered.

With regard to the second category which is related to dynamic number of thelight sources, the concept used is the same as the shadow’s LOD concept. Whenthe object is moved further away, the number of light sources used to generate theshadows decreases. This means that the light source changes dynamically accordingto the distance of a virtual object. Thus, the farther the position of the objects fromthe observer’s position, the lower level of details on the shadows of objects and lightsources are used. This approach is used such that the performance of the soft shadowscreation becomes more efficient. Figure 7 is the result of QSI using DML techniquein virtual environments.

4 Experiment

This section discusses the experiment of rendering soft shadows in AR environments.The experiments are conducted using three layers where every layer has a differentnumber of hard shadow samples. The sample layers involved are 3, 5 and 7 layers,respectively. It means the algorithm is tested with three different types of case studiesinclude 3, 5 and 7 layers. In each layer the number of hard shadow samples is differentwhich are 8, 16 and 32. The purpose of using three different samples of layers inthis research is to facilitate evaluation of the quality of soft shadows produced bythe number of layers used. Each sample resulted in different shadow qualities. Inthis experiment, tests are conducted using a computer and camera device with thefollowing stipulated specifications.

The following result (tables) of an object with 69,666 triangles is obtained usingan AMD Athlon 64 X2 6400+ Windsor 3.2GHz Socket AM2 125W Dual-Core withGraphic Hardware GeForce 7025 NVIDIA nForce 630a. The specifications of the

Fig. 7 a Soft shadows with 8 hard shadow samples, b soft shadow generated using QSI and64 samples

Multimed Tools Appl

Camera are Aloha Digital PC Camera with VGA, 30 Frame/Sec Frame rate, up to 8megapixels, Glass element lens.

Before performing the tests, determination of parameters which needed to beconstant must have been known. This is necessary for conducting the evaluationprocess so that it becomes balanced. The parameters that are constant in this researchare computer and camera specifications and testing environment. This experiment isconducted in a dark room with one light source coming from the real environment.

5 Results

This section reveals the results from the experiment. The results are discussed basedon two indicators including the quality of soft shadows generated from the threelevels of details and the performances which are measured based on Frame perSecond (FPS).

In this research, a virtual shadow produced by a virtual object is directly com-parable with the shadows of real objects. It is essential to compare the quality ofvirtual shadows with the quality of shadows of the real objects. In addition, thevirtual shadow generated in this research is based on the directions of real light

Fig. 8 Soft shadows using LOD

Multimed Tools Appl

Fig. 9 Soft shadows using QSI and LOD

sources. Therefore, virtual soft shadows produced in this research have the samedirections as those of the real objects. Figure 8 shows the results of soft shadows inAR environments where the QSI is not applied yet.

Figure 9 consists of 7 layers of hard shadow using QSI technique. The result isacceptable compared to the previous results. The virtual shadows produced by avirtual object are directly comparable with those of real objects.

The experiment measures the performance of the system based on FPS. Figure 10shows a graph comparing the FPS among the three samples of the shadows witha number of light sources used in this experiment. The maximum value of FPSthat is set on this camera is as high as 15 FPS with which the performed tests aresynchronized. FPS levels shown in this graph depend on the number of light sourceswhich are used. The samples with 3 layers of the image reaching the highest FPS of 15FPS use as many as 4 and 8 light sources. Accordingly, the graph shows the value ofthe FPS going down with the addition of number of light sources as every light sourceneeds the rendered shadows. Thus, a higher number of light sources for generatingsoft shadows and DML are used to avoid sustaining additional costs compared withthe ordinary algorithm. In the remaining parts of the paper, QSI is applied with anobject with 69,666 numbers of triangles.

The graph shows how the use of light sources for shadow creation affects the valueof the FPS system. Based on this graph, the first sample with three layers of hardshadows produces higher FPS values compared with the second and third samples.In fact, in the case of more shadow layers the quality of shadows increases while

Fig. 10 FPS versus thenumber of light sourcesfor three image layers

Multimed Tools Appl

FPS values decrease. This is in contrast with the number of layers of shadows, inwhich a slightly lower quality of shadows is produced, whereas a higher FPS value isinvolved. To reduce the computational costs of shadow creation, this research carriesout level of details of shadows. Each level of detail involves virtual objects that havedifferent numbers of vertices depending on the details of a predetermined distance.Accordingly, the further the objects are from the camera position, the less details ofvirtual object will be used to produce these shadows.

Figure 11a above shows the graph of FPS values obtained from the results ofshadows LOD for soft shadows, in which three samples of image layers are used. Asdiscussed previously, this research involves three levels of the detail. Based on thegraph above, the FPS shows an increasing trend, when the number of light sourcesto produce soft shadows is reduced. The third level of LOD shows a higher rateof FPS up to 15 FPS. This is because the number of vertices of a virtual object islower than the number of vertices of a virtual object on the first and the second level.Figure 11b and c show the graphs of FPS which are based on the results of shadowsLOD for soft shadows using five and seven samples of image layers, respectively. Theresults represent similar trends with that of graph shown in Fig. 11a. In fact, the FPSvalues increase when the number of light sources is reduced. The performance of thesystem is still lower even though the LOD is applied. To increase the performance, adynamic light source is used to create the soft shadows based on the distance of theshadows from the user’s view. Figure 12 shows the results of FPS values when shadowLOD is applied with dynamic light sources for three different samples of layers.

Based on the results obtained from the graph in Fig. 12a, it can be seen thatthe third level of LOD shows a higher FPS, which is 15 FPS for all the tested

Fig. 11 FPS versus the number of light sources of shadow LOD with three layers (a), five layers (b)and seven layers (c) of images

Multimed Tools Appl

Fig. 12 FPS versus the number of light sources for shadow LOD and dynamic light source with threelayers a, five layers b and seven layers c of images

number of light sources. With regard to generating soft shadows, the LOD for thefirst and the second level shows gradual decrease as the number of light sourcesincrease. Figure 12b and c show gradual decrease of FPS as the number of lightsources for three levels of LOD increase. This is when five and seven layers areused, respectively. Comparing the results from Fig. 12, it is evident that Fig. 12band c have a lower FPS, which is due to the use of an increasing number of layersof hard shadows. However, performance increases with shadow LOD and dynamiclight sources compared with the case having only the shadow LOD. In conclusion,the experimental results of this research show that samples with three layers using 32light sources are sufficient to produce realistic soft shadows in AR environments.

6 Discussion

Although integration of hard shadow samples could produce soft shadows, increasingthe number of light sources leads to a decrease in the FPS. Therefore, the proposedQSI technique and the three layers of hard shadows for each light source aresufficient. The quality of soft shadows is measured based on direct comparison withreal shadows. The performance of soft shadow creation is measured by the number offrames per second. Testing performance is important in the AR environment becausethe AR technology involves the real environment. This test is also intended to ensurethe development of AR applications in achieving the standard rate for a particularAR application. The performance of samples of hard shadow layers for soft shadowcreation is also measured.

Multimed Tools Appl

Fig. 13 Different amount of penumbra

A DML technique with dynamic real light sources is used to reduce the com-putation costs of soft shadow creation. In this technique, three levels of detail areperformed on the shadows, and the light source is based on a specified distance. Thus,the farther the object from the camera position, the lower the details of shadows onobjects and light sources used. In addition, the number of polygons at each level ofdetail influences the cost of the system. QSI has enhanced the quality of soft shadows.In addition, when combined with DML, it is the final desired condition to generatesoft shadows in AR. Based on the results obtained from these tests, the number oflight sources needed for the production of soft shadows is sufficient at 32. In addition,having implemented a detailed multi-layer and used a dynamic real light source, thevalue of the FPS has been enhanced for the system. As a result, it can be seen thatthis technique improves performance of soft shadow generation.

Fig. 14 Shadow generation in AR. a Result of convolution shadow maps [2], b Castro results [6],c soft shadow generation using QSI and LOD

Multimed Tools Appl

Fig. 15 Soft shadows using QSI and LOD with same shapes using real light source

A simple comparison of the latest soft shadow works in AR such as Aittala [2]and Castro et al. [6] with the present work can be observed in Fig. 14. In addition todesirable rendering time due to using projection shadows, the quality of soft shadowsis acceptable. Figure 14a shows the results of CoSMs in AR performed by Aittala [2].Figure 14b is the result of Castro et al’s work, while (c) is the result of the presenttechnique using QSI and LOD. Soft penumbra and inadequate umbra demonstratethe high performance quality of the proposed technique to generate soft shadowsin AR. The other advantage of our technique is controllable amount of penumbrawhich as shown in Fig. 13. This ability helps users use this technique for indoor andoutdoor rendering where the amount of penumbra may change. Semi-soft shadowsconstructed with less penumbra are the types of shadows which can be employed foroutdoor rendering.

Figure 15 shows the soft shadows generated using QSI and LOD with the sameshape to reveal the ability of our approach. In these pictures the virtual green boxand the virtual cylinder appear adjacent to the real ones. As Fig. 15 illustrates theproposed technique could be performed for real light source by replacing the lightsource tracking with manual virtual light source which can be manually placed onthe real light source. The amount of umbra and penumbra in Fig. 15 is similar to thereal umbra and penumbra cast by real objects. This technique not only prevents anincrease in rendering time but also enhances shadow quality. Nevertheless, the onlylimitation with our approach is self-shadowing and casting shadows on other objectsdue to using projection shadows which can be solved by replacing the projectionshadows with shadow maps or any other improvements on shadow maps such asCascaded Shadow Maps [8] and Hybrid Shadow Maps [26].

The controllable amount of penumbra due to possible managing of the light sourcepositions with respect to the enhancement on soft shadow quality can be consideredas an advantage of our work. Maintaining the balance between soft shadow quality incomplex objects and FPS is another contribution of this study. This claim is depictedin Figs. 13, 14 and 15.

7 Conclusion and future work

This research has implemented soft shadows in AR environments. Compared to hardshadows, the purpose of soft shadows is to add realistic features on shadows in ARenvironments. This is because of the fact that hard shadows still have drawbacks

Multimed Tools Appl

such as having sharp and hard-edged outlines, which are considered to be deficientin the appearance of shadows. The creation of soft shadows is based on the reallight source in which the direction of virtual shadows is similar to the direction ofreal shadows in the real environments. The technique used to obtain the real lightsource position is based on the median cut algorithm which was developed by otherresearchers. The marker which plays the role of the light source is constructed usinga reflective sphere in glossy black paint and a 2D marker [7, 20, 22]. To improve theimplementation process of soft shadows, QSI technique is applied to enhance thequality. Furthermore, DML is employed with dynamic light sources to complete thebalance of trade-off between quality and FPS.

In conclusion, this research has resulted in the creation of realistic soft shadowsbased on real light source direction. In addition to enhancing the soft shadow qual-ity, computational costs are reduced. Computational cost reduction is essential toenhancing performance such that the appearance of AR environments can be morerealistic. In future works, the techniques to generate soft shadows can be improvedbased on system requirements. AR in outdoor rendering environments based on reallight source from the sun could be more beneficial for mixed reality environments.In addition, virtual shadows produced in AR environments correspond to those ofreal objects and vice versa. To be more realistic in the context of AR environments,virtual objects in this environment must be illuminated with real lighting so as toappear as real objects.

Acknowledgements This research was supported by Vot. J13000.7282.4F085 FRGS grant at theUTM VicubeLab, Department of Computer Graphics and Multimedia, Faculty of Computing,Universiti Teknologi Malaysia.

References

1. Agusanto K, Li L, Chuangui Z, Sing N (2003) Photorealistic rendering for augmented real-ity using environment illumination. In: Proceedings of the 2nd IEEE and ACM internationalsymposium on mixed and augmented reality, pp 208–216

2. Aittala M (2010) Inverse lighting and photorealistic rendering for augmented reality. Vis Comput26(6–8):669–678

3. Annen T, Dong Z, Mertens T, Bekaert P, Seidel HP Kautz J, (2008) Real-time, all-frequencyshadows in dynamic scenes. In: ACM transactions on graphics (TOG), vol 27, no 3. ACM, p 34

4. Azuma R (1997) A survey of augmented reality. Presence: Teleoperators and Virtual Environ-ments 6:355–385

5. Carmigniani J et al (2011) Augmented reality technologies, systems and applications. MultimTools Appl 51(1):341–377

6. Castro T, Figueiredo L, Velho L (2012) Realistic shadows for mobile augmented reality.In: Virtual and augmented reality (SVR). 2012 14th symposium, pp 36–45

7. Debevec P (2005) A median cut algorithm for light probe sampling. In: International conferenceon computer graphics and interactive techniques, Los Angeles, California

8. Dimitrov R (2007) Cascaded shadow maps. Developer documentation, NVIDIA Corp9. Gibson S, Murta A (2000) Interactive rendering with real world illumination. In: Proceedings

of eurographics symposium on rendering 2000, pp 365–37610. Gibson S, Cook J, Howard T, Hubbold R (2003) Rapid shadow generation in real-world lighting

environments. In: Proceedings of eurographics symposium on rendering 2003, pp 219–22911. Haller M (2004) Photorealism or/and non-photorealism in augmented reality. In: Proceedings

of the 2004 ACM SIGGRAPH international conference on virtual, reality continuum and itsapplications in industry. ACM, pp 189–196

Multimed Tools Appl

12. Haller M, Drab S, Hartmann W (2003) A real-time shadow approach for an augmented realityapplication using shadow volumes. In: Proceedings of VRST 03, pp 56–65

13. Hartmann W, Zauner J, Haller M, Luckeneder T, Woess W (2003) Shadow catcher: a visionbased illumination condition sensor using ARtoolkit. In: 2003 IEEE internation augmentedreality toolkit workshop (IEEE Cat. No.03EX780), pp 44–55

14. Hasenfratz J, Lapierre M, Holzschuch N, Sillion F (2003) A survey of real-time soft shadowsalgorithms. Eurographics 22(4):753–774

15. Hensley J, Scheuermann T, Coombe G, Singh M, Lastra A (2005) Fast summed-area tablegeneration and its applications. Comput Graph Forum 24(3):547–555

16. Herf M, Heckbert PS (1996) Fast soft shadows. In: Visual proceedings SIGGRAPH 96,p 145

17. Hu B, Brown C (2005) Cast shadows in augmented reality systems. PhD thesis, University ofRochester, Department of Computer Science

18. Hughes CE, Konttinen J, Pattanaik SN (2004) The future of mixed reality: Issues in illumina-tion and shadows. In: The interservice/industry training, simulation and education conference(I/ITSEC), vol 2004, no 1. National Training Systems Association

19. Jacobs K, Angus C, Loscos C (2005) Automatic generation of consistent shadows for augmentedreality. In: Proceedings graphics interface, Vancouver, Canada

20. Jensen BF, Laursen JS, Madsen JB, Pedersen TW (2009) Simplifying real time light sourcetracking and credible shadow generation for augmented reality. Institute for Media Technology,Aalborg University

21. Kanbara M, Yokoya N (2002) Geometric and photometric registration for real-time augmentedreality. In: IEEE and ACM international symposium on mixed and augmented reality (ISMAR),p 279

22. Kanbara M, Yokoya N (2004) Real-time estimation of light source environment for photo-realistic augmented reality. In: Proceedings of the 17th international conference on patternrecognition. Cambridge, UK, pp 911–914

23. Kolivand H, Sunar SM (2012) An overview on based real-time shadow techniques in virtualenvironment. Telkomnika 10(1):171–178

24. Kolivand H, Sunar MS (2013) Covering photo-realistic properties of outdoor compo-nents with the effects of sky color in mixed reality. Multimed Tools Appl. doi:10.1007/s11042-013-1494-9

25. Kolivand H, Sunar SM (2013) A survey on volume shadows in computer graphics. IETE TechRev 30(1):38–46

26. Kolivand H, Sunar MS (2012) Real-time outdoor rendering using hybrid shadow maps. Int JInnov Comput Appl Inf Control (IJICIC) 8(10):7169–7184

27. Kolivand H, Amirshakarami A, Sunar MS (2011) Real-time projection shadow with respect tosun’s position in virtual environments. Int J Comput Sci Issues 8(6):80–84

28. Krevelen D, Poelman R (2010) A survey of augmented reality technologies, applications andlimitations. Int J Virtual Reality 9(2):1–20

29. Lensing P, Broll W (2012) Instant indirect illumination for dynamic mixed reality scenes. In:Mixed and augmented reality (ISMAR). 2012 IEEE international symposium, pp 109–118

30. Madsen CB, Laursen R (2007) A scalable GPU-based approach to shading and shadowing forphoto-realistic real-time augmented reality. In: Proceedings international conference on graphicstheory and applications. Barcelona, Spain, pp 252–261

31. Madsen CB, Nielsen M (2008) Towards probe-less augmented reality. A position paper,Computer Vision and Media Technology Lab, Aalborg University, Aalborg, Denmark

32. Madsen CB, Sorensen MKD, Vittrup M (2003) The important of shadows in augmented reality.In: Proceedings 6th annual international workshop on presence, Aalborg, Denmark

33. Naemura T, Nitta T, Mimura A, Harashima H (2002) Virtual shadows in mixed reality environ-ment using flashlight-like devices. Trans Virtual Reality Society of Japan 7(2):227–237

34. Nakano G, Kitahara I, Ohta Y (2008) Generating perceptually-correct shadows for mixed re-ality. In: 7th IEEE/ACM international symposium on mixed and augmented reality, pp 173–174

35. Poupyrev I, Tan DS, Billinghurst M, Kato H, Regenbrecht H, Tetsutani N (2002) Developing ageneric augmented-reality interface. Computer 35(3):44–50

36. Sato I, Sato Y, Ikeuchi K (1999) Acquiring a radiance distribution to superimpose virtual objectsonto a real scene. IEEE Trans Vis Comput Graph 5(1):1–12

37. Sood R (2012) A 3D augmented reality model viewer. In: Pro android augmented reality,pp 159–220

Multimed Tools Appl

38. Sugano N, Kato H, Tachibana K (2003) The effects of shadow representation of virtual objectsin augmented reality. In: IEEE/ACM international symposium on mixed and augmented reality(ISMAR 2003). IEEE Computer Society, pp 76–83

39. Yan F (2008) Estimation of light source environment for illumination consistency of augmentedreality. In: First international congress on image and signal processing, pp 771–775

40. Yeoh RC, Zhou SZ (2009) Consistent real-time lighting for virtual objects in augmented reality.In: 8th IEEE international symposium on mixed and augmented reality (ISMAR 2009), Orlando,USA, pp 223–224

Hoshang Kolivand is now pursuing PhD in UTM ViCubelab under the supervision of Dr MohdShahrizal Bin Sunar. His research interests include Computer Graphics and Virtual Environment.Received the M.S degree in Applied Mathematics and computer from Amirkabir University, Iranin 1999. B.S degree in Computer Science & Mathematic from Islamic Azad University, Iran in1997. Previously he was a lecturer in Shahid Beheshti University Iran. He has published enormousarticles in international journals and conferences as well as national journals, conference proceedingsand technical papers including article in a book. Hoshang Kolivand is an active reviewer ofsome conferences and international journals. He had published many books in object-orientedprogramming and one in mathematics.

Zakiah Noh is master’s student in computer graphics and multimedia at the Universiti TeknologiMalaysia. She holds degree in computer science from the Universiti Teknologi Malaysia in 2008 anddiploma holders in computer science (information system) in 2006 at the same university. Zakiah has2 years of experience in Augmented Reality fields and joined several conference at the internationaland local seminar.

Multimed Tools Appl

Mohd Shahrizal Sunar obtained his PhD from National University of Malaysia in 2008. His majorfield of study is real-time and interactive computer graphics and virtual environment. He received hisMSc in Computer Graphics and Virtual Environment (2001) from The University of Hull, UK andBSc degree in Computer Science majoring in Computer Graphics (1999) from Universiti TeknologiMalaysia. He served as academic member at Computer Graphics and Multimedia Department,Faculty of Computer Science and Information System, Universiti Teknologi Malaysia since 1999.Since 2009, he had been given responsibility to lead the department. The current research programthat he lead are Driving Simulator, Augmented Reality, Natural Interaction and Creative ContentTechnology. He had published numerous articles in international as well as national journals,conference proceedings and technical papers including article in magazines. Dr. Shahrizal is an activeprofessional member of ACM SIGGRAPH. He is also a member Malaysian Society of Mathematicsand Science.