Collaborative Augmented Reality Approach For Multi-user Interaction in Urban Simulation

Ajune Wanis IsmailDepartment of Computer Graphic and Multimedia,

Faculty of Computer Science and Information System Universiti Teknologi Malaysia 81310 Skudai Johor, Malaysia

ajune@utm.my

Mohd Shahrizal SunarDepartment of Computer Graphic and Multimedia,

Faculty of Computer Science and Information SystemUniversiti Teknologi Malaysia 81310 Skudai Johor, Malaysia

shah@fsksm.utm.my

Abstract—Augmented reality (AR) environment allows user or multi-user to interact with 2D and 3D data. AR simply can provide a collaborative interactive AR environment for urban simulation, where users can interact naturally and intuitively. AR collaboration approach can be effectively used to develop face to face interfaces. This is because AR provides seamless interaction between real and virtual environments, the ability to enhance reality, the presence of spatial cues for face-to-face and remote collaboration, support of a tangible interface metaphor, the ability to transition smoothly between reality and virtuality. The fusion between real and virtual world, existed in AR environment, achieves higher interactivity as a key features of collaborative AR. Collaborative AR approach allows multi-user to simultaneously share a real world surrounding them and a virtual world. The features of collaboration in AR environment will be identified. The key for the proposed technique of precise registration between both worlds and multi-user are emphasized for the collaborations using AR environment. Common problems in AR environment and issues in collaborative AR will be discussed. This paper will give an overview for collaborative AR framework employed in urban simulation and the multi-user interaction on how to share these virtual spaces with other users in collaboration. The work will also cover numerous systems in different cases of collaborative AR environments for multi-user interaction.

Keywords— Augmented Reality, Multi-User Interaction, Collaborative, Urban Simulation.

I. INTRODUCTION

Augmented reality (AR) environments are defined by Milgram and Kishino [1] as those in which real world and virtual world objects are presented together on a single display. The AR applications have shown that AR interfaces can enable a person to interact with the real world in ways never before possible [2]. A comprehensive survey of AR is found in [3].

Architectural of urban simulation for urban planning and design process has always caught up with highly cooperative tasks. Individual phases within a project often change between close cooperative situations, for instance, during design and review meetings, and individual work carried out by the users or third parties. Real collaboration is often limited to the very early design sketches. From an architect’s

point of view it would be desirable to have an additional support tool allowing enhancing the cooperation in a way that supports real collaboration within meetings. As a result, this would in turn allow for much faster design and review cycles.

Fig. 1 A concept of Mixed Reality introduced by Milgram and Kishinowith illustrated the new appearance of urban planning from traditional view

of blocks to virtual reality.

Traditionally urban design is perceived, communicated and created using physical and digital media. However, these realms are handled as separate entities, which hinder collaboration, understanding and communication. Collaborative AR systems can integrate these tasks into one media type and allow a different conversation with complex issues. Human Computer Interfaces and Tangible User Interfaces play a key role in AR. They allow the combination with both the real and virtual component of an urban design project.

A collaborative AR system supporting for urban simulation is an environment and tools to support the collaboration between the experts involved in these particular meetings [4]. AR collaboration and multi-users for urban simulation are reflected by the use of intuitive interaction mechanisms, which allow even untrained users to benefit from the enhancements provided by the AR environment. Additionally, a few current AR systems for urban simulation exist with tools such as CAD systems and simulation programs that already used by the people involved. They were integrated to provide a rather seamless transition between their individual daily work and the collaborative work at the round table meetings. However, in developing AR environment for urban simulation, a few common problems might be encountered.

2009 International Conference on Information and Multimedia Technology

978-0-7695-3922-5/09 $26.00 © 2009 IEEE

DOI 10.1109/ICIMT.2009.68

19

Therefore, in this paper we are focusing in two problems, there are registrations and user collaboration. The main challenge in AR applications for urban simulation is to ensure, that the displayed information is aligned with the user’s view of the surrounding world, because the human visual system is very sensitive to misalignment. This alignment is often referred to as registration. Related to registration is the problem of calibrating a head-worn display and dealing with slight shifts of the HMD relative to the eye which result in noticeable misalignment. The collaboration between users in virtual spaces in AR environment for urban simulation has been an important research topic in the past few years.

The paper will also cover numerous systems of collaborative AR environments for multi-user. There have been a number of previous works can be found in this research. Studierstube [5] is an environment for collaboration AR which uses HMD displays to project artificial 3D objects into the real world. The Augmented Round Table, as mentioned in [6], is providing a new interface for collaborative design and review for architecture and urban planning. It focuses on providing an intuitive environment, which supports natural interaction with virtual objects while sustaining accessible collaboration and interaction mechanisms.

As stated in [7] the MagicMeeting system limits direct user communication due to the use of video augmentation. In [8], Grasset et. al. proposed MARE system that uses wired tracking devices and non-commodity hardware. The Luminous Table highlighted in [9] that attempts to tackle this issue by integrating multiple forms of physical and digital representations. 2D drawings, 3D physical models, and digital simulation are overlaid into a single information space in order to support the urban design process. In [10], ARTHUR allows user directly access to geometrical data and provides a tabletop immersive environment. Seichter [11] claimed in his research that one of the most important purposes of collaborative AR environment is to promote social interaction among multi-user located in the same physical space.

II. COLLABORATIVE AUGMENTED REALITY

ENVIRONMENT

In face-to-face conversation, when people talk with each other, while they are collaborating on a real world task, there is a dynamic and easy interchange of focus between the shared workspace and the speakers’ interpersonal space. The shared workspace is the common task area between collaborators, while the interpersonal space is the common communications space [13]. The interfaces may not be used if they provide the same experience as face-to-face communication; they must enable users to go “beyond being there” and enhance the collaborative experience [14, 15]. Considering face-to-face interaction as a specific type of communications medium, it becomes apparent that this approach requires one medium to adapt to another, pitting the strengths of face-to-face collaboration against other interfaces [14]. In fact, because of the nature of the medium,

it may be impossible for mediated collaborations to provide the same experience as face-to-face collaboration [14] [16].

A. Features of collaborative AR environment

The multi-user AR environment developed allows multiple users to share a virtual space projected into their common working environment [6]. The features included in collaborative environment as below:

1) Virtuality : The potential of objects can be viewed and examined, either are not reachable or do not exist in the physical spaces can be carried out in AR environment.

2) Augmentation : It allows a smooth fusion between real objects and virtual properties in propose processes.

3) Multi-user : The situations where multi-user gather together in a room to discuss, design, or perform other types of cooperative work or collaboration. Higher interactivity among multi-user effectively interacts with themselves using normal sense of human interactions, like verbal and gestures are now simply possible in AR setup.

4) Independence : Each user has the option to move freely control his own independent viewpoint and interaction can also be performed independently without interrupting any action that performed by other users.

5) Individuality : Models and objects are shared among users means all users can observe the same coherent model, consistent in sense of visibility.

B. Issues in Collaborative Augmented Reality

This subsection addresses issues in order to employ collaborative AR environment. The issues listed as below:

1) RegistrationRegistration is common issue in AR. As already

mentioned in section 1, registration error is misalignment that the displayed information is aligned with the user’s view of the surrounding world, because the human visual system is very sensitive to misalignment [25, 28]. As stated in Azuma [2] and Grasset [8], they classified the possible alignment errors in two categories:

Static error : The misalignment the user can perceive when he is not moving and remain static. Static error is due to various causes: position error, noise, numerical error, and any possible causes since AR system is very sensitive to their misalignment.

Dynamic error : The misalignment the user can perceive when he is moving around in the environment. Dynamic error is due to the time lag between the objects or user’s movement and visual feedback.

2) Interaction techniqueInteractions can be characterized into two groups: user-

object interactions and user-user interactions. User-user interactions deal with communication between users, such as chat. As classified by Grasset and friends research [8], they also have highlighted the issues in interacting with real

20

objects in AR environment. The issues mentioned in their paper are:

Occlusion compatibility: It happens between virtual and real object which is parts of virtual objects behind by real objects should not be displayed.

Law of physics: It happens between the two world which is virtual object laid on top of real object should remain in line view.

Manipulation sensitivity: The awareness on the AR system which is it must be updated after a real object is moved in the shared space.

3) User Callibration technique An ideal calibration method is able to separate user-

specific parameters (eye position) from parameters that are constant for agiven HMD or tracker sensor setup, minimizing the number of datapoints necessary for user-specific calibration.

III. COLLABORATIVE AR ENVIRONMENT FOR MULTI-USER IN URBAN SIMULATION

This section summarizes collaborative AR systems that employs collaborative AR systems have already developed and proposed by group of AR researcher that focusing on AR collaboration for multi-user in urban studies and planning, as shown in Table I.

TABLE IAR SYSTEMS THAT SUPPORT COLLABORATION FOR MULTI-USER IN

COLLABORATIVE AR

AR System System Overview2002 : The

Luminous TableIt attempts to tackle this issue by integrating multiple forms of physical and digital representations. 2D drawings, 3D physical models, and digital simulation are overlaid into a single information space in order to support the urban design process. [9]

2003 : The Augmented

Round Table

Provides a new interface for collaborative design and review for architecture and urban planning. Focus on providing an intuitive environment, which supports natural interaction with virtual objects while sustaining accessible collaboration and interaction mechanisms. [6]

2004 : ARTHUR It allows designers a direct access to geometrical data and provides a tabletop immersive environment. The trackers to allow for a natural 3D collaboration. However, it attempts to integrate manipulation techniques of CAD directly through spatial 2D menus and does not provide an estimate of this approach. [10]

2005 : Magic Meeting

A Collaborative Tangible Augmented Reality System. Supports product review meetings by augmenting a real meeting location. Instead of real mock-ups, virtual 3D models are used, which may be loaded into the environment from usual desktop applications or from PDAs.[7]

2007 : Urban Sketcher

A mix of multi-modal input devices enhances collaborative interaction in real-time, while visual feedback is given to multi-users on a projected live video augmentation from urban sketcher. Sketching, modifying the scene on site. [23]

I. MULTI-USER INTERACTION TECHNIQUE FOR URBAN

SIMULATION IN COLLABORATIVE AR

AR can create an augmented workspace which multi-user may access a shared space populated by virtual objects, while remaining grounded in the real world [23]. One of the most important purposes of collaborative AR environment is to promote social interaction among multi-user located in the same physical space [24]. This is powerful for urban studies and planning purposes when multi-user are collocated and can use natural means of communication such as speech or gestures.

In this paper we have addressed a several problems occur in collaborative AR environment for urban simulation. Then we focused the highlighted issues on multi-users in collaborative AR in order to design and construct the collaborative AR technique for urban simulation. We want to bring collaborative AR for multi-users into the urban simulation and provide new effective approach in urban simulation and to enhance collaboration shared physical urban workspace in planning and design process.

AR collaboration and multi-user for urban simulation are reflected by the use of intuitive interaction mechanisms, which allow even untrained users to benefit from the enhancements provided by the AR environment. However, in this paper we discussed two problems in collaborative AR; there are registrations and multi-user collaboration. The main challenge in applying AR for urban simulation is to ensure, that the displayed information is aligned with the user’s view of the surrounding world, because the human visual system is very sensitive to misalignment means aligning the virtual information with the real world so that it appears to be a part of the real environment. This registration must occur in real time; otherwise the user will experience an unacceptable lag of the visualization when he moves. Another issue will occur is how to share these virtual spaces with other users in collaboration. AR environment allows multiple participants or multi-user to interact with 2D and 3D data. AR simply can provide a collaborative interactive AR environment for urban simulation, where users can interact naturally and intuitively. In addition, the collaborative AR makes multi-user in urban simulation to share simultaneously a real world and virtual world.

A. Collaborative AR environment for Urban Simulation

Firstly there is a need to construct the framework of the collaborative AR environment. All the display technologies that suitable for AR environment should be setting up. Marker in augmented reality is assigned as base marker and also can be used as interaction tools which we can assign action for each single marker. The base marker is a set of marker used to station the virtual object [27, 28].

21

1) Setting collaborative AR workbench: Once we got a few users interact among them in AR environment with user-object interaction and user-user interaction, we referred this kind of environment as collaborative AR environment. The framework for collaborative AR environment is setting up as illustrated in figure 3.

Fig. 3 Collaborative AR environment framework

2) Marker: We may treat each of these markers as a single marker. Each of these markers can be assigned with an action. We may use the pattern type provided with ARToolKit library. However we customize the patterns so that it meets your preferences.

3) Shared communication cues: Shared communication cues in the space between them as shown in figure 4 below. These cues include gaze, gesture, and nonverbal behaviors.

Fig.4 Structure of shared communication cues

B. Multi-user Interaction Technique

Interaction is defined as mutual or reciprocal action or influence. In real world everything and everybody interacts with everybody else. The physical interaction changes the physical properties of an object, like shape, size, position and material properties. Several methods have been developed for interactions inside a homogenous Virtual Environment, and between an application and its user. Adaptation of 2D techniques in 3D, Usage of new input devices and trackers,

Direct manipulation, Physical mnemonics, and Gesture actions

However, these techniques have some limitations: which no reference to actual working habits in a real environment, lack of haptic input and feedback, and cumbersome input devices [27]. There is clear need for a natural way of interacting with and designing of a virtual environment. ARenvironment will allow users to be in a real environment and perceive the world as it is. Our approach is that using AR environments and virtual humans as a mediator between a real environment and a virtual environment; we can achieve a natural way of interaction between human and machine. A virtual object in urban simulation can act very real [27, 28].

1) Marker as interaction tools: Marker can be used as interaction tools. The pattern for these markers must be unique. The interaction marker set consists of a few markers. Interaction markers are used to manipulate object projected on this base marker set. Both markers must work together to make the whole environment works.

2) Gestures interaction: We can define interaction with a virtual object in urban simulation as response actions such as gestures or verbal output from a participant. Gestures in collaborative AR consist with three categories as a pointing gestures, a pick whereas a gesture that involved picking or moving an object and the third is a collaborative gesture such as passing an object from one person to the other

3) Multi-user interaction: We can define how many users or participants that willing to perform in our workbench. Consider gestures or markers as interaction tools, algorithms need to be constructed in order to overcome registration and tracking errors. Sensing and bandwidth measurement problems also need to be solved for user collaboration. To overcome these problems, we need to make adjustment or corrections for camera orientation and shared objects position. Time updated and measurement also should be performed for sensing and bandwidth errors.

IV. CONCLUSIONS

In this paper we performed our research on multi-user interaction technique in collaborative AR for urban simulation. On the first section of this paper, we have described about AR environment and AR for urban simulation. Next section we identified the collaborative AR environment. We found common features in AR collaborative environment. In this paper also we described the issues in AR collaboration system. This paper hassummarized the previous collaborative AR environments that have already developed and proposed by group of AR researcher that focusing on AR collaboration for multi-user.

AR techniques can be effectively used to develop fundamentally different interfaces for face-to-face and remote collaboration. This is because AR provides seamless interaction between real and virtual environments, the ability

22

to enhance reality, the presence of spatial cues for face-to-face and remote collaboration, support of a tangible interface metaphor, the ability to transition smoothly between reality and virtuality. In this paper we have provided several examples of collaborative AR environment on previous works of collaborative AR environment. Finally we have focused the highlighted the framework and technique for multi-user interaction. Then final section has discussed about the interaction technique for multi-user in collaborative AR environment. The challenge of this paper is to bring collaborative AR for multi-user into the urban simulation in order to provide new effective approach in urban simulationand to enhance collaboration shared physical urban workspace. As conclusion, we hope that this research is useful for AR urban simulation and also brings benefit to the computer graphics community especially in AR emerging technologies.

REFERENCES

[1] Milgram, P and Kishino, F., “A taxonomy of Mixed Reality visual displays”, IEICE Transactions on Information Systems E77-D, 12. (1994)

[2] Azuma R., “A Survey of Augmented Reality. PRESENCE: Teleoperators and Virtual Environments”, Vol. 6, No. 4, pp. 355-385. (1997)

[3] Azuma R., Baillot Y., Behringer R., Feiner S., Julier S. and Blair M., “Recent advances in augmented reality. IEEE Computer Graphics and Applications”, 20-38. (2001).

[4] Fuhrmann A., Loffelmann H., and Schmalstieg D., 1997, “Collaborative Augmented Reality: Exploring Dynamical Systems,” Proc. Visualization '97, pp.459- 462.

[5] Schmalstieg, D., Fuhrmann, A., Szalavari, Z., Gervautz, M., 1996, “Studierstube - An Environment for Collaboration in Augmented Reality”. In CVE ’96 Workshop Proceedings, 19-20th September 1996, Nottingham, Great Britain. (1996)

[6] Broll W., Stoerring M. & Mottram C,“The Augmented Round Table – a new Interface to Urban Planning and Architectural Design” Published by IOS Press, (c) IFIP, 2003, pp. 1103-1104 Human-Computer Interaction ─ INTERACT’03 M. Rauterberg et al. (Eds.) Published by IOS Press, (c) IFIP, pp. 1103-1104,(2003)

[7] Regenbrecht, H, Wagner, M and Baratoff, G., “MagicMeeting: A Collaborative Tangible Augmented Reality System”, Virtual Reality 6(3): 151-166. (2004)

[8] Grasset R. and J.D. Gascuel. “MARE: Multiuser Augmented Reality Environment on table setup”. In SIGGRAPH, Conference Abstracts and Applications, Computer Graphics Annual Conference Series. (2002).

[9] Ishii H., Ben-Joseph E., Underkoffler J., Yeung L., Chak D., Kanji Z., and Piper B., 2002, Augmented Urban Planning Workbench: Overlaying Drawings, Physical Models and Digital Simulation, Proceedings of the IEEE/ACM International Symposium on Mixed and Augmented Reality ISMAR 2002, IEEE Computer Society, ISBN 0-7695-1781-1, pp. 203–211. (2002)

[10] Broll W., Lindt I., Ohlenburg J., Wittk M. Yuan C., Novotny T, Fatah, Schiecky, Mottramy C., Strothmann A., 2004, “ARTHUR: A Collaborative Augmented Environment for Architectural Design and Urban Planning”, Journal of Virtual Reality and Broadcasting, Volume 1, no. 1, page 1, (2004).

[11] Seichter H., "Augmented Reality And Tangibility In Urban Design”, A Dong, A Vande Moere & JS Gero (eds), CAADFutures’07,Springer 3–16.(2007)

[12] Billinghurst M, Kato H., and Poupyrev I., “The Magic-Book –Moving Seamlessly between Reality and Virtuality”, IEEE Computer

Graphics and Applications 21 (2001), no. 3, 6–8, ISSN 0272-1716. (2001)

[13] Ishii, H., Kobayashi, M., Arita, K., 1994, “Iterative Design of Seamless Collaboration Media”. Communications of the ACM, Vol 37, No. 8, August 1994, pp. 83-97. (1994)

[14] Ajune Wanis, Mohd Shahrizal Sunar, “Multi-user Interaction in Collaborative Augmented Reality for Urban Simulation”, The International Conference on Graphic and Image Processing 2009 (ICGIP’09), Nov. 13-15, 2009, Kota Kinabalu Sabah, Malaysia. (2009)

[15] Zakiah Noh, Ajune Wanis, Mohd Shahrizal Sunar, “Exploring the potential of using Augmented Reality Approach in Cultural Heritage System.”, 2nd International Conference on Advanced Computer Theory and Engineering (ICACTE’09), Sept. 25-27 2009, Cairo, Egypt. (2009)

[16] Ajune Wanis, Mohd Shahrizal Sunar , “An Overview On Collaborative Augmented Reality Environment Potentials In Urban Planning ”, 5th International Conference on Information & Communication Technology and Systems (ICTS’09), 4th August 2009, Surabaya Indonesia. (2009)

[17] Cheok, A. D., Yang, X., Zhou, Z. Y., Billinghurst, M.,and Kato, H., “Touch-Space: Mixed Reality Game Space Based on Ubiquitous, Tangible, and Social Computing. Personal and Ubiquitous Computing”, vol. 6, no. 2, 430–442. (2002).

[18] Ohshima, T., Sato, K., Yamamoto, H., Tamura, H.,1998, “AR2Hockey: A case study of collaborative augmented reality”, In Proceedings of VRAIS'98, IEEE Press: Los Alamitos, pp.268-295. (1998)

[19] Thomas B. H. ,Close B., Donoghue J., Bondi P., Morris M.,Wayne P., “ARQuake: An Outdoor/Indoor Augmented Reality First Person Application”. Proceedings of the Fourth International Symposium on Wearable Computers (ISWC'00), October 2000, Atlanta USA. (2000)

[20] Szalavári Zs., Gervautz, M., “The Personal Interaction Panel – A Two-Handed Interface for Augmented Reality”, Computer Graphics Forum (Proceedings of EUROGRAPHICS’97) 16(3), September 1997, pp. 335-346. (1997)

[21] Wagner, D, Pintaric,T, Ledermann,F and Schmalstieg, D 2005, ‘Towards Massively Multi-User Augmented Reality on Handheld Devices,’ Vienna University of Technology. (2005)

[22] Ajune W., Zakiah N., “Augmented Reality and applications”, Advances in Computer Graphic & Virtual Environment, Shahrizal M. & Najib M. (Eds), Universiti Teknologi Malaysia, Pages 87 – 105. (2008)

[23] Schmalstieg D., Sareika M. "Urban Sketcher: Mixing Realities in the Urban Planning and Design Process", In Proceedings of 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, Pages: 27-30. (2007)

[24] Seichter H., "Augmented Reality And Tangibility In Urban Design”, A Dong, A Vande Moere & JS Gero (eds), CAADFutures’07,Springer 3–16.(2007)

[25] Ajune Wanis., Mohd Shahrizal Sunar, "Survey on Collaborative AR for Multi-user in Urban Studies and Planning” Edutainment 2009 Springer-Verlag Berlin Heidelberg, Lecture Note in Computer Science LNCS 5670. ( 2009)

[26] Ajune Wanis., Mohd Shahrizal Sunar, "Survey on Collaborative AR for Multi-user in Urban Simulation” 5th Postgraduate Annual Research Seminar 2009 (PARS'09), Johor:Universiti Teknologi Malaysia. (2009)

[27] Sato K, Ban Y, Chihara K. MR Aided Engineering: Inspection Support Systems Integrating Virtual Instruments and Process Control. In “Mixed Reality Merging Real and Virtual Worlds”. Ed. By Y. Ohta and H. Tamura. Springer 1999. ISBN 3-540-65623-5. (1999)

[28] Simon G, Lepetit V, Berger MO. Computer Vision Methods for Registration: Mixing 3D Knowledge and 2D Correspondence for Accurate Image Composition. Proceedings of the IWAR’98, San Francisco, Nov. 1, 1998. A.K.Peters. (1999)

23