Multi-user Interaction in Collaborative Augmented Reality for Urban Simulation

Ajune Wanis Ismail, Mohd Shahrizal SunarDepartment of Computer Graphic and Multimedia,

Faculty of Computer Science and Information SystemUniversiti Teknologi Malaysia81310 Skudai Johor, Malaysia

ajunewanis@ymail.comshah@fsksm.utm.my

Abstract— Augmented reality (AR) environment allows useror multi-user to interact with 2D and 3D data. AR simply canprovide a collaborative interactive AR environment for urbansimulation, where users can interact naturally and intuitively.AR collaboration approach can be effectively used to developdifferent interfaces for face-to-face and remote collaboration.This is because AR provides seamless interaction between realand virtual environments, the ability to enhance reality, thepresence of spatial cues for face-to-face and remotecollaboration, support of a tangible interface metaphor, theability to transition smoothly between reality and virtuality. Inaddition, the collaborative AR makes multi-user in urbansimulation to share simultaneously a real world and virtualworld. The fusion between real and virtual world, existed inAR environment by see-through HMDs, achieves higherinteractivity as a key features of collaborative AR. In real-time,precise registration between both worlds and multi-user arecrucial for the collaborations. Collaborative AR approachallows multi-user to simultaneously share a real worldsurrounding them and a virtual world. Common problems inAR environment will be discussed and major issues incollaborative AR will be explained details in this survey. Thefeatures of collaboration in AR environment are will beidentified and the requirements of collaborative AR will bedefined. This paper will give an overview on collaborative ARenvironment for multi-user in urban studies and planning.The work will also cover numerous systems of collaborativeAR environments for multi-user.

Keywords— Augmented Reality, Multi-User, Collaborative,Urban Simulation.

I. INTRODUCTION

Augmented reality (AR) environments are defined byMilgram and Kishino [1] as those in which real world andvirtual world objects are presented together on a singledisplay. The AR applications have shown that ARinterfaces can enable a person to interact with the real worldin ways never before possible [2]. A comprehensive surveyof AR is found in [3].

Architectural of urban simulation for urban planning anddesign process has always caught up with highlycooperative tasks. Individual phases within a project oftenchange between close cooperative situations, for instance,during design and review meetings, and individual workcarried out by the users or third parties. Real collaboration

is often limited to the very early design sketches. From anarchitect’s point of view it would be desirable to have anadditional support tool allowing enhancing the cooperationin a way that supports real collaboration within meetings.As a result, this would in turn allow for much faster designand review cycles.

Fig. 1 A concept of Mixed Reality introduced by Milgram and Kishinowith illustrated the new appearance of urban planning from traditional

view of blocks to virtual reality.

Traditionally urban design is perceived, communicatedand created using physical and digital media. However,these realms are handled as separate entities, which hindercollaboration, understanding and communication.Collaborative AR systems can integrate these tasks into onemedia type and allow a different conversation with complexissues. Human Computer Interfaces and Tangible UserInterfaces play a key role in AR. They allow thecombination with both the real and virtual component of anurban design project.

A collaborative AR system supporting for urbansimulation is an environment and tools to support thecollaboration between the experts involved in theseparticular meetings [4]. AR collaboration and multi-usersfor urban simulation are reflected by the use of intuitiveinteraction mechanisms, which allow even untrained usersto benefit from the enhancements provided by the ARenvironment. Additionally, a few current AR systems forurban simulation exist with tools such as CAD systems andsimulation programs that already used by the peopleinvolved. They were integrated to provide a rather seamlesstransition between their individual daily work and thecollaborative work at the round table meetings. However, in

2009 Second International Conference on Machine Vision

978-0-7695-3944-7 2010

U.S. Government Work Not Protected by U.S. Copyright

DOI 10.1109/ICMV.2009.40

309

developing AR environment for urban simulation, a fewcommon problems might be encountered.

Therefore, in this paper we are focusing in two problems,there are registrations and user collaboration. The mainchallenge in AR applications for urban simulation is toensure, that the displayed information is aligned with theuser’s view of the surrounding world, because the humanvisual system is very sensitive to misalignment. Thisalignment is often referred to as registration. Related toregistration is the problem of calibrating a head-worndisplay and dealing with slight shifts of the HMD relative tothe eye which result in noticeable misalignment. Thecollaboration between users in virtual spaces in ARenvironment for urban simulation has been an importantresearch topic in the past few years.

The research will also cover numerous systems ofcollaborative AR environments for multi-user. There havebeen a number of previous works can be found in thisresearch. Studierstube [5] is an environment forcollaboration AR which uses HMD displays to projectartificial 3D objects into the real world. The AugmentedRound Table, as mentioned in [6], is providing a newinterface for collaborative design and review forarchitecture and urban planning. The interface relies onunobtrusive input mechanisms and natural and intuitive userinteractions. It focuses on providing an intuitiveenvironment, which supports natural interaction with virtualobjects while sustaining accessible collaboration andinteraction mechanisms.

As stated in [7] the MagicMeeting system limits directuser communication due to the use of video augmentation.In [8], Grasset et. al. proposed MARE system that useswired tracking devices and non-commodity hardware. TheLuminous Table highlighted in [9] that attempts to tacklethis issue by integrating multiple forms of physical anddigital representations. 2D drawings, 3D physical models,and digital simulation are overlaid into a single informationspace in order to support the urban design process. In [10],ARTHUR allows user directly access to geometrical dataand provides a tabletop immersive environment. Seichter[11] claimed in his research that one of the most importantpurposes of collaborative AR environment is to promotesocial interaction among multi-user located in the samephysical space.

.

II. COLLABORATIVE AUGMENTED REALITY

In face-to-face collaboration, people use speech, gesture,gaze and non-verbal cues to attempt to communicate in theclearest possible fashion. Real objects are also more thanjust a source of information; they are also the constituentsof the collaborative activity, create reference frames forcommunication and alter the dynamics of interaction,especially in multi-user settings. The multi-user ARenvironment developed allows multiple users to share avirtual space projected into their common working

environment [6]. Features of collaborative AR environmentas below:

A. VirtualityThe potential of objects can be viewed and examined,

either are not reachable or do not exist in the physicalspaces can be carried out in AR environment.

B. AugmentationReal objects that do exist in real world can be augmented

by virtual annotations. As a result, it allows a smooth fusionbetween real objects and virtual properties in proposeprocesses.

C. Multi-user.The situations where multi-user gather together in a room

to discuss, design, or perform other types of cooperativework or collaboration. Higher interactivity among multi-user effectively interacts with themselves using normalsense of human interactions, like verbal and gestures arenow simply possible in an augmented reality setup.

D. Independence.Each user has the option to move freely and

independently of the other users. Each user may freelycontrol his own independent viewpoint. However, not onlyis observation independent, interaction can also beperformed independently without interrupting any actionthat performed by other users.

E. Individuality.In general models and objects are shared among users

means all users can observe the same coherent model,consistent in sense of visibility. The displayed objects canalso be different for each observer, as required by theapplication’s needs and the individual’s option.

Fig. 2 Features of Collaborative AR

III. MAJOR ISSUES IN COLLABORATIVE AUGMENTED REALITY

AR is ideal for collaborative interfaces because itaddresses two major issues in collaborative AR:seamlessness and enhancing reality [12].

A. Seamlessness

310

The shared workspace is the common task area betweencollaborators, while the interpersonal space is the commoncommunications space. In [13], seams can be of two types:Functional Seams: Discontinuities between differentfunctional workspaces, user is forcing to change modes ofoperation. Cognitive Seams: Discontinuities betweenexisting and new work practices, user is forcing to be taughtnew ways of working.

B. Enhancing RealityRemoving the seams in a collaborative interface is not

enough. The interfaces may not be used if they provide thesame experience as face-to-face communication; they mustenable users to go “beyond being there” and enhance thecollaborative experience [14, 15]. Considering face-to-faceinteraction as a specific type of communications medium, itbecomes apparent that this approach requires one mediumto adapt to another, pitting the strengths of face-to-facecollaboration against other interfaces [14]. In fact, becauseof the nature of the medium, it may be impossible formediated collaborations to provide the same experience asface-to-face collaboration [14] [16].

IV. COLLABORATIVE AR ENVIRONMENT FOR MULTI-USER

The section before we have discussed on features ofcollaborative AR environment that have a force onovercoming major issues with collaborative AR. Heredescribes the previous collaborative AR environments thathave already developed and proposed by group of ARresearcher that focusing on AR collaboration for multi-user.As shown in Table I, the comparative studies on previousworks of AR projects, focusing on collaborative ARenvironment for multi-user.

TABLE IAR SYSTEMS THAT SUPPORT COLLABORATION FOR MULTI-USER IN

COLLABORATIVE AR

AR System System Overview

ARTHURA Collaborative

AugmentedEnvironment for

ArchitecturalDesign and Urban

Planning.

It allows designers a direct access togeometrical data and provides a tabletopimmersive environment. By using opticalaugmentation and wireless computer–visionbased trackers to allow for a natural 3Dcollaboration. Virtual objects are displayedusing stereoscopic visualization to seamlesslymix them into the real world. However, itattempts to integrate manipulation techniquesof CAD directly through spatial 2D menusand does not provide an estimate of thisapproach. [10]

The LuminousTable

Augmented UrbanPlanning

Workbench

“Luminous Table” that attempts to tackle thisissue by integrating multiple forms ofphysical and digital representations. 2Ddrawings, 3D physical models, and digitalsimulation are overlaid into a singleinformation space in order to support theurban design process. The physical objectsare tracked with cameras. It was conceived as a platform for multi-layered physical and

digital representations. [9]

The AugmentedRound Table

A new Interfaceto Urban Planningand Architectural

Design

Augmented Round Table is providing a newinterface for collaborative design and reviewfor architecture and urban planning. Theinterface relies on unobtrusive inputmechanisms and natural and intuitive userinteractions. Focus on providing an intuitiveenvironment, which supports naturalinteraction with virtual objects whilesustaining accessible collaboration andinteraction mechanisms. [6]

AR2 Hockey

AR2Hockey (Augmented Reality AiR Hockey). System where players can share a physical game field, mallets, and a virtualpuck to play an air-hockey game. [18]

ARQuake

A Quake game in AR. In Quake, the playerruns around a virtual world, shooting atmonsters, collecting objects, and completingobjectives. [19]

Virtual Roulett

A multi-user augmented gamingenvironment, Virtual Roulett, a famouscasino game shows the table with multi-userare playing. [20]

The InvisibleTrain

Towards Massively Multi-User AR onHandheld Devices. The Invisible Train is a mobile, collaborative multi-user AR game,virtual trains on a real wooden tiny rail track.[21, 22]

V. MULTI-USER INTERACTION TECHNIQUE FOR URBANSIMULATION IN COLLABORATIVE AR

AR can create an augmented workspace which multi-usermay access a shared space populated by virtual objects,while remaining grounded in the real world [23]. One of themost important purposes of collaborative AR environmentis to promote social interaction among multi-user located inthe same physical space [24]. This is powerful for urbanstudies and planning purposes when multi-user arecollocated and can use natural means of communicationsuch as speech or gestures.

AR Collaboration technique can be effectively used todevelop fundamentally different interfaces for face-to-faceand remote collaboration. This is because AR providesseamless interaction between real and virtual environments,the ability to enhance reality, the presence of spatial cuesfor face-to-face and remote collaboration, support of atangible interface metaphor, the ability to transitionsmoothly between reality and virtuality [25, 26]. In thispaper we have addressed a several problems occur incollaborative AR environment for urban simulation. Thenwe focused the highlighted issues on multi-users incollaborative AR in order to design and construct thecollaborative AR technique for urban simulation. We wantto bring collaborative AR for multi-users into the urbansimulation and provide new effective approach in urbansimulation and to enhance collaboration shared physicalurban workspace in planning and design process.

AR collaboration and multi-user for urban simulation arereflected by the use of intuitive interaction mechanisms,

311

which allow even untrained users to benefit from theenhancements provided by the AR environment. However,in this paper we discussed two problems in collaborativeAR; there are registrations and multi-user collaboration.The main challenge in applying AR for urban simulation isto ensure, that the displayed information is aligned with theuser’s view of the surrounding world, because the humanvisual system is very sensitive to misalignment. Thisalignment is often referred to as registration. Related toregistration is the problem of calibrating a head-worndisplay and dealing with slight shifts of the HMD relative tothe eye which result in noticeable misalignment. One of themost relevant technical issues in developing ARapplications is solving the registration problem: registrationmeans aligning the virtual information with the real worldso that it appears to be a part of the real environment. Thisregistration must occur in real time; otherwise the user willexperience an unacceptable lag of the visualization when hemoves. Another issue will occur is how to share thesevirtual spaces with other users in collaboration. ARenvironment allows multiple participants or multi-user tointeract with 2D and 3D data. AR simply can provide acollaborative interactive AR environment for urbansimulation, where users can interact naturally andintuitively. In addition, the collaborative AR makes multi-user in urban simulation to share simultaneously a realworld and virtual world.

A. Collaborative AR environment for Urban SimulationIn collaboration, people use speech, gesture, gaze and

non-verbal cues to attempt to communicate in the clearestpossible fashion. Interactions can be characterized into twogroups: user-object interactions and user-user interactions.User-user interactions deal with communication betweenusers, such as chat. Firstly there is a need to construct theframework of the collaborative AR environment. All thedisplay technologies that suitable for AR environmentshould be setting up. Marker in augmented reality isassigned as base marker and also can be used as interactiontools which we can assign action for each single marker.The base marker is a set of marker used to station thevirtual object [27, 28].

Fig. 3 Collaborative AR environment framework

Setting collaborative AR workbench: Once we got a fewusers interact among them in AR environment with user-object interaction and user-user interaction, we referred thiskind of environment as collaborative AR environment. Theframework for collaborative AR environment is setting upas illustrated in figure 3.

Marker: We may treat each of these markers as a singlemarker. Each of these markers can be assigned with anaction. We may use the pattern type provided withARToolKit library. However we customize the patterns sothat it meets your preferences.

Shared communication cues: Shared communication cuesin the space between them. These cues include gaze, gesture,and nonverbal behaviors.

B. Multi-user InteractionInteraction is defined as mutual or reciprocal action or

influence. In real world everything and everybody interactswith everybody else. However, these techniques have somelimitations: which no reference to actual working habits in areal environment, lack of haptic input and feedback, andcumbersome input devices [27]. AR environment will allowusers to be in a real environment and perceive the world asit is. We can achieve a natural way of interaction betweenhuman and machine. We can define how many users orparticipants that willing to perform in our workbench.Consider gestures or markers as interaction tools,algorithms need to be constructed in order to overcomeregistration and tracking errors. Sensing and bandwidthmeasurement problems also need to be solved for usercollaboration. To overcome these problems, we need tomake adjustment or corrections for camera orientation andshared objects position. Time updated and measurementalso should be performed for sensing and bandwidth errors.[27, 28].

Marker as interaction tools: Marker can be used asinteraction tools. The pattern for these markers must beunique. The interaction marker set consists of a fewmarkers. Interaction markers are used to manipulate objectprojected on this base marker set. Both markers must worktogether to make the whole environment works.

Gestures interaction: We can define interaction with avirtual object in urban simulation as response actions suchas gestures or verbal output from a participant. Gestures incollaborative AR consist with three categories, first is apoint that pointing gestures, a pick that a gesture involvedpicking or moving an object and the third is a collaborativegesture such as passing an object from one person to theother.

VI. CONCLUSIONS

In this paper we performed our survey on multi-userinteraction technique in collaborative AR for urbansimulation. On the first section of this paper, we have

312

described about AR environment and AR for urbansimulation. Next section we identified the collaborative ARenvironment. We found common features in ARcollaborative environment. In this paper also we describedthe major issues in AR collaboration system. This paper hassummarized the previous collaborative AR environmentsthat have already developed and proposed by group of ARresearcher that focusing on AR collaboration for multi-user.Users experience with these interfaces have shown that theyfacilitate collaboration in a natural manner, enabling peopleto use normal gestures and non-verbal behavior in face-to-face collaboration, and to have access to their conventionaltools and workplace by both face-to-face and remotecollaboration.

AR techniques can be effectively used to developfundamentally different interfaces for face-to-face andremote collaboration. This is because AR provides seamlessinteraction between real and virtual environments, theability to enhance reality, the presence of spatial cues forface-to-face and remote collaboration, support of a tangibleinterface metaphor, the ability to transition smoothlybetween reality and virtuality. In this paper we haveprovided several examples of collaborative ARenvironment in gaming and also comparative studies onprevious works of collaborative AR environment for urbanstudies and planning. Finally we have focused thehighlighted the framework and technique for multi-userinteraction. Then final section has discussed about theinteraction technique for multi-user in collaborative ARenvironment. The challenge of this survey is to bringcollaborative AR for multi-user into the urban simulation inorder to provide new effective approach in urban simulationand to enhance collaboration shared physical urbanworkspace. As conclusion, we hope that this research isuseful for AR urban simulation and also brings benefit tothe computer graphics community especially in ARemerging technologies.

ACKNOWLEDGMENT

We would like express our appreciation to UniversitiTeknologi Malaysia (UTM) and Malaysian Ministry ofScience, Technology and Innovation (MOSTI) forproviding financial support of this research througheScienceFund (79282) grant scheme.

REFERENCES[1] Milgram, P and Kishino, F., “A taxonomy of Mixed Reality visual

displays”, IEICE Transactions on Information Systems E77-D, 12.(1994)

[2] Azuma R., “A Survey of Augmented Reality. PRESENCE:Teleoperators and Virtual Environments”, Vol. 6, No. 4, pp. 355-385. (1997)

[3] Azuma R., Baillot Y., Behringer R., Feiner S., Julier S. and Blair M.,“Recent advances in augmented reality. IEEE Computer Graphicsand Applications”, 20-38. (2001).

[4] Fuhrmann A., Loffelmann H., and Schmalstieg D., 1997,“Collaborative Augmented Reality: Exploring Dynamical Systems,”Proc. Visualization '97, pp.459- 462.

[5] Schmalstieg, D., Fuhrmann, A., Szalavari, Z., Gervautz, M., 1996,“Studierstube - An Environment for Collaboration in AugmentedReality”. In CVE ’96 Workshop Proceedings, 19-20th September1996, Nottingham, Great Britain. (1996)

[6] Broll W., Stoerring M. & Mottram C,“The Augmented Round Table– a new Interface to Urban Planning and Architectural Design”Published by IOS Press, (c) IFIP, 2003, pp. 1103-1104 Human-Computer Interaction ─ INTERACT’03 M. Rauterberg et al. (Eds.)Published by IOS Press, (c) IFIP, pp. 1103-1104,(2003)

[7] Regenbrecht, H, Wagner, M and Baratoff, G., “MagicMeeting: ACollaborative Tangible Augmented Reality System”, Virtual Reality6(3): 151-166. (2004)

[8] Grasset R. and J.D. Gascuel. “MARE: Multiuser AugmentedReality Environment on table setup”. In SIGGRAPH, ConferenceAbstracts and Applications, Computer Graphics Annual ConferenceSeries. (2002).

[9] Ishii H., Ben-Joseph E., Underkoffler J., Yeung L., Chak D., KanjiZ., and Piper B., 2002, Augmented Urban Planning Workbench:Overlaying Drawings, Physical Models and Digital Simulation,Proceedings of the IEEE/ACM International Symposium on Mixedand Augmented Reality ISMAR 2002, IEEE Computer Society,ISBN 0-7695-1781-1, pp. 203–211. (2002)

[10] Broll W., Lindt I., Ohlenburg J., Wittk M. Yuan C., Novotny T,Fatah, Schiecky, Mottramy C., Strothmann A., 2004, “ARTHUR: ACollaborative Augmented Environment for Architectural Designand Urban Planning”, Journal of Virtual Reality and Broadcasting,Volume 1, no. 1, page 1, (2004).

[11] Seichter H., "Augmented Reality And Tangibility In Urban Design”,A Dong, A Vande Moere & JS Gero (eds),CAADFutures’07,Springer 3–16.(2007)

[12] Billinghurst M, Kato H., and Poupyrev I., “The Magic-Book –Moving Seamlessly between Reality and Virtuality”, IEEEComputer Graphics and Applications 21 (2001), no. 3, 6–8, ISSN0272-1716. (2001)

[13] Ishii, H., Kobayashi, M., Arita, K., 1994, “Iterative Design ofSeamless Collaboration Media”. Communications of the ACM, Vol37, No. 8, August 1994, pp. 83-97. (1994)

[14] Hollan, J., Stornetta, S., 1992, “Beyond Being There”. InProceedings of CHI ’92, 1992, New York: ACM Press, pp.119-125.(1992)

[15] Fish, R.S., Kraut, R.E., Root, R.W., Rice, R., “Evaluating Video asa Technology for Informal Communication. Bellcore TechnicalMemorandum. (1991)

[16] Gaver, W. The Affordances of Media Spaces for Collaboration, inProc. CSCW’92, Toronto, Canada, Oct.31-Nov.4, 1992, New York:ACM Press, pp. 17-24. (1992)

[17] Cheok, A. D., Yang, X., Zhou, Z. Y., Billinghurst, M.,and Kato, H.,“Touch-Space: Mixed Reality Game Space Based on Ubiquitous,Tangible, and Social Computing. Personal and UbiquitousComputing”, vol. 6, no. 2, 430–442. (2002).

[18] Ohshima, T., Sato, K., Yamamoto, H., Tamura, H.,1998,“AR2Hockey: A case study of collaborative augmented reality”, InProceedings of VRAIS'98, IEEE Press: Los Alamitos, pp.268-295.(1998)

[19] Thomas B. H. ,Close B., Donoghue J., Bondi P., Morris M.,WayneP., “ARQuake: An Outdoor/Indoor Augmented Reality First PersonApplication”. Proceedings of the Fourth International Symposiumon Wearable Computers (ISWC'00), October 2000, Atlanta USA.(2000)

[20] Szalavári Zs., Gervautz, M., “The Personal Interaction Panel – ATwo-Handed Interface for Augmented Reality”, Computer GraphicsForum (Proceedings of EUROGRAPHICS’97) 16(3), September1997, pp. 335-346. (1997)

[21] Wagner, D, Pintaric,T, Ledermann,F and Schmalstieg, D 2005,‘Towards Massively Multi-User Augmented Reality on HandheldDevices,’ Vienna University of Technology. (2005)

[22] Ajune W., Zakiah N., “Augmented Reality and applications”,Advances in Computer Graphic & Virtual Environment, ShahrizalM. & Najib M. (Eds), Universiti Teknologi Malaysia, Pages 87 –105. (2008)

313

[23] Schmalstieg D., Sareika M. "Urban Sketcher: Mixing Realities inthe Urban Planning and Design Process", In Proceedings of 6thIEEE and ACM International Symposium on Mixed andAugmented Reality, Pages: 27-30. (2007)

[24] Seichter H., "Augmented Reality And Tangibility In Urban Design”,A Dong, A Vande Moere & JS Gero (eds),CAADFutures’07,Springer 3–16.(2007)

[25] Ajune Wanis., Mohd Shahrizal Sunar, "Survey on Collaborative ARfor Multi-user in Urban Studies and Planning” Edutainment 2009Springer-Verlag Berlin Heidelberg, Lecture Note in ComputerScience LNCS 5093. ( 2009)

[26] Ajune Wanis., Mohd Shahrizal Sunar, "Survey on Collaborative ARfor Multi-user in Urban Simulation” 5th Postgraduate AnnualResearch Seminar 2009 (PARS'09), Johor:Universiti TeknologiMalaysia. (2009)

[27] Sato K, Ban Y, Chihara K. MR Aided Engineering: InspectionSupport Systems Integrating Virtual Instruments and ProcessControl. In “Mixed Reality Merging Real and Virtual Worlds”. Ed.By Y. Ohta and H. Tamura. Springer 1999. ISBN 3-540-65623-5.(1999)

[28] Simon G, Lepetit V, Berger MO. Computer Vision Methods forRegistration: Mixing 3D Knowledge and 2D Correspondence forAccurate Image Composition. Proceedings of the IWAR’98, SanFrancisco, Nov. 1, 1998. A.K.Peters. (1999)

314