http://excel.fit.vutbr.cz

Combining Spatial Augmented Reality Interfacewith Mixed Reality Head-mounted Display toEnhance User Understanding of Robotic ProgramsDaniel Bambusek*

AbstractThis paper focuses on usability of mixed reality head-mounted display – Microsoft HoloLens – inhuman-robot collaborative workspace – ARTable. Use of the headset is demonstrated by createduser interface which helps regular workers to better and faster understand the ARTable system. Itallows to spatially visualize learned programs, without the necessity to run the robot itself. Theuser is guided by 3D animation of individual programs and by device voice, which helps him toget a clear idea of what will happen if he runs the program directly on the robot. This approachcould be also helpful when the user wants to verify that he programed the robot as he wanted orjust wants to quickly overview what will be the result of the current program. Using mixed realitydisplays enables to visualize valuable spatial information, such as robot perception.

Keywords: ARTable — Microsoft HoloLens — Augmented Reality — mixed reality head-mounteddisplay — human-robot interaction — PR2 — program visualization — human-robot collaborativeworkspace

Supplementary Material: Demonstration Video — Downloadable Code*xbambu04@stud.fit.vutbr.cz, Faculty of Information Technology, Brno University of Technology

1. Introduction

As industrial collaborative robots are getting more af-fordable, it is likely that small and medium enterpriseswill soon adopt such robots in order to increase pro-ductivity. However, in such enterprises, productionbatches are smaller and products may be customizedfor a specific contract. This requires reprogrammingrobots for particular tasks, which could be challengingdue to necessity of robot-specific knowledge. Thus itwould be beneficial to enable ordinary-skilled workerto program these robots easily. Therefore a proto-type of a human-robot collaborative workspace – theARTable was created (see Figure 1), which presents a

novel approach to programming robots based on cog-nition, spatial augmented reality and multimodal inputand output [1].

Programming of collaborative robots in this ap-proach is based on setting up parameters for specificinstructions which forms the whole program. Theseinstructions are described only with text identifiers andas experiments showed up, users had difficulties inorienting in such program topology. There were uncer-tainties of how instructions follow, what exactly willspecific program do or what is expected from user toprogram.

This work aims to clarify user’s uncertainties inprogramming robots by providing program visualiza-

tion system in spatial augmented reality with use ofMicrosoft HoloLens. Being able to see what will hap-pen when specific program runs, what specific instruc-tion means and what is expected from the user to pro-gram should rapidly help in understanding of the wholeARTable system. Proposed program visualization isfully controllable with voice or projected buttons, soeven a user familiar with the ARTable might find itusable, when he can validate correctness of a new pro-gram or quickly visualize an existing one and see it’soutcome by skipping to the end, without the necessityof running it on the robot itself.

2. Background

There are several approaches of end-user robot pro-gramming. One of the main techniques are Program-ming by Demonstration and Visual Programming. Bothapproaches are combined in a robot programming sys-tem – Code3 [2], which enables a non-roboticist pro-grammers to create a complex robot programs usinglandmarks detection, programming by demonstrationand visual drag-and-drop programming interface thatlets users define control flow logic of their programs.

Usage of augmented reality (AR) in industrialrobotics may lead to decreased workload through-out robot control by showing useful information tousers [3]. A concept and architecture for programmingindustrial robots with use of AR is presented in [4].In [5], authors developed an application that augmentsan industrial robot for shop floor tasks and evaluateddifferent approaches of augmentation – using mobilephone or smart glasses (Epson Moverio). Glasses suf-fered from need of continuous marker tracking andlimited field of view. However, usage of hand-helddevices might be limiting because it prevents usage ofboth hands.

Use of the mixed reality head-mounted displaysfrees user’s hands and as Rosen et al. [6] proved, 3Dvisualization of valuable information is more effectivethan 2D visualization. They used Microsoft HoloLensto visualize robot arm motion intent and asked partici-pants to determine which robot motions are going tocollide with obstacles on the table. They reached upto 16% increase in accuracy and 62% decrease in timeit took participants to complete the task when usingHoloLens.

3. The ARTable – Augmented RealityCollaborative Workspace

As the title suggests, the term “ARTable” stands for aprototype of a human-robot collaborative workspace

Figure 1. Setup of the human-robot collaborativeworkspace – the ARTable. In this example the usersets program parameters using robot’s arm andgestures.

based on augmented reality. It was created by the re-search group Robo@FIT1 at Brno University of Tech-nology. The prototype focuses on small and mediumenterprises where it should enable ordinary-skilledworker to program a robot on a high level of abstrac-tion and perform collaborative tasks effectively andsafely. It aims to show possibilities of the collaborationbetween human and robot in near future [1].

The source code and technical documentation ofthis prototype is available at github2.

3.1 Core ComponentsThe ARTable setup uses the PR2 robot from WillowGarage, which is located behind the table (from user’sperspective) and serves as a demonstrator of a near-future collaborative robot. However, the setup is robotindependent.

The table itself has a touch-sensitive layer ontowhich an interface is projected using Acer P6600 pro-jector mounted on the construction above the table.The projected interface contains various elements tovisualize state of the robot and current task. Togetherwith the touch-sensitive layer, the table serves as themain source of input for the system and feedback forthe user. Another possible way of passing input intothe system is by direct manipulation with robotic arm.The table is complemented with speakers on each sides,which serves as supplementary source of feedback forthe user.

1http://www.fit.vutbr.cz/research/groups/robo/index.php.en

2https://github.com/robofit/artable

The system uses two Kinects version 2, mountedon a stable tripod on each side of the table and oneKinect version 1, mounted on the head of the PR2, fora detection and tracking of objects in the scene. Eachtripod has it’s own processing unit (Intel NUC) wherethe projector and sensors are connected. This units areconnected to the central computer. AR codes, attachedto objects, are used for better tracking. They are alsoused for calibration of the whole system. Regarding tosoftware setup, the ARTable uses the Robot OperatingSystem (ROS).

3.2 Supported Programs and InstructionsPrograms are sets of instructions which are collectedinto blocks. These instructions are linked togetherbased on the result of specific instruction (success orfailure). Currently, the system supports parametricinstructions as pick from polygon (to pick an objectfrom specified polygon on the table), pick from feeder(to pick an object from gravity feeder), place to pose(to place previously picked object to selected place onthe table) and apply glue (to simulate gluing). Usersets parameters in these instructions, such as objecttype to pick, place position of picked object or robotsgripper position for picking objects from feeder.

Besides mentioned instructions, there are also fewnon-parametric instructions: get ready (which movesrobot arms to a default position), wait for user (whichpauses the program execution if user moves away fromthe table) and wait until user finishes (which pausesthe program execution until user finishes current inter-action with objects on the table).

4. Integration of Microsoft HoloLens intothe ARTable System

Microsoft HoloLens is a headset for mixed reality, de-veloped by Microsoft. It’s wireless, runs Windows 10,has precise spatial mapping [7], and is capable of 3ways of interaction – gaze, gestures and voice. De-vice draws computer generated holograms into the realworld and has a potential to be used in many scenar-ios, for instance in medicine, to provide doctors with“X-ray vision” during surgery [8], in education, in ar-chitecture, to overview buildings in 3D model size [8]or just to augment skilled workers and technicians inthe field to help the users with tasks in which they lackexperiences [9]. However, the device is suffering withreally small field of view – 30◦ x 17.5◦ degrees (aspectratio 16:9) [10], which is a limitation that negativelyaffects experience with use in ARTable system.

In order to successfully integrate the HoloLens intothe ARTable system there were two major steps to take

Figure 2. Left: Calibration of the HoloLens withrespect to the ARTable. Colored cube visualizes howwell the calibration was done and has an option buttonfor recalibration. Red side of the cube representspositive x axis, green side represents positive y axisand blue positive z axis. Right: Marker used forcalibration acquired from ARToolKit.

– make communication between these two systemswork and calibrate the HoloLens with respect to theARTable. For development, the Unity3 was chosen,which is a cross-platform game engine used mainly fordeveloping 3D and 2D video games or simulations.

4.1 Communication via rosbridgeThe ARTable system runs on ROS with Ubuntu butHoloLens runs on Windows. So there is a problem ofhow to transport data from one operating system toanother. This problem is solved by ROS tool – ros-bridge, which is a tool that provides a JSON API toROS functionality for non-ROS programs.

For such communication I used and extended ex-isting Unity library – ROSBridgeLib4 (author MichaelJenkin, edited by Mathias Ciarlo Thorstensen). Thislibrary uses SimpleJSON5 parser for parsing JSONmessages. However, due to differences in scriptingbackends of Unity editor (Mono) and UWP applica-tions (.NET), functionality of this library had to beextended to support the UWP format which is usedby HoloLens. With inspiration from Unity libraryholoROS6, created by Gabriel Santos Solia, who suc-cessfully integrated Microsoft HoloLens with ROS viarosbridge, in order to simulate a holographic turtlesim7

environment, I was able to extend the ROSBridgeLibfunctionality to support communication between theARTable (ROS) and Unity along with HoloLens (Win-dows).

Having this type of communication enables to con-nect multiple headsets with the ARTable, where all ofthem receive same data.

3https://unity3d.com/4https://github.com/MathiasCiarlo/

ROSBridgeLib5https://github.com/Bunny83/SimpleJSON6https://github.com/soliagabriel/holoROS7http://wiki.ros.org/turtlesim

Figure 3. Spectator’s view of the workspace extendedby virtual holograms seen through the headset. In thissituation the user observes visualization of “Stoolassembly” program.

4.2 HoloLens Calibration with Respect to theARTable

The ARTable is calibrated with use of three markersplaced into corners of the table. Origin of coordinatesystem is estimated in bottom left corner of the table.The HoloLens also needs to know where that corner is.This is accomplished by detection of a marker placedin that spot with use of HoloLens world-facing camera(see Figure 2). For detection purposes, the HoloLens-ARToolKit8 was used, which is a Unity library thatintegrates ARToolKit9 functionality with UWP appli-cations, created by Long Qian.

If marker detection is successful, colored cube isdrawn on this marker spot which helps to visualizehow well the calibration was done. This cube is af-terwards wrapped with an empty Unity gameObjectwhich is then stored as a spatial anchor and set as aparent to every other visual object in the scene. Thusit represents origin of the table. Spatial anchors arepersistent during sessions, so calibration process needsto be done only once. There is also possibility of recal-ibration by clicking the Recalibration button as can beseen in Figure 2.

After calibration, one last step has to be made.There is a conflict in coordinate systems, the ARTableuses right-handed while Unity and therefore HoloLensuses left-handed. To solve this problem, every objectposition and rotation that came from ARTable needsto invert it’s y axis and Euler x and z angle.

8https://github.com/qian256/HoloLensARToolKit

9https://github.com/artoolkit/artoolkit5

5. User Interface for Robot Programs Vi-sualization

Current 2D projected user interface is limited in visu-alization of spatial information. For instance, it drawsbounding boxes around detected objects placed on thetable, that indicates what robot actually sees. However,sensors are able to detect these objects even if they arenot placed directly on table, e.g. when they are in grav-ity feeder. Also the way it displays set parameters fora given program is not intuitive enough. Users were of-ten uncertain in how instructions follow, what exactlywill specific program do or what they are supposed toprogram.

Provided solution eliminates limitations of 2D pro-jected interface. With use of the Microsoft HoloLens,drawing of 3D spatial bounding boxes around detectedobjects is easily solved. In order to eliminate user’suncertainties in programming the robot, programs vi-sualization mode was implemented. This mode ex-tends current 2D projected interface by Visualize but-ton which starts the visualization itself. Only pro-grams with set parameters are able to visualize (seeFigure 4). After hitting this button, the interface checksif HoloLens are running, otherwise visualization won’tstart. Visualization is fully controllable with providedbuttons – Pause/Resume, Stop and Replay (as shownin Figure 4).

As described in subsection 3.2, ARTable programsare compounded of various instructions that are linkedtogether. Instruction set of currently visualized pro-gram is loaded from ROS environment into HoloLenswhich then dynamically builds this program from inter-nally implemented classes representing these instruc-tions. With this approach any program variations builtfrom supported instructions should visualize correctly.After build, execution starts.

Example usage of the headset is shown in Figure 3,where the user observes visualization of “Stool assem-bly” program. Visualization of this program is furtherdemonstrated in Figure 5. Whole process visualizes in-dividual instructions which are connected together in acontext of the program. For example, when visualizingpick from feeder instruction, virtual object is created,which is then picked by a virtual robot gripper from thefeeder and placed on the preset position on the table.This object then stays active in the scene in case thatanother instruction could want to manipulate with it(e.g. to apply glue to it or to move it somewhere else).Whole process is commented by speech synthesizer.Besides provided buttons for controlling the visualiza-tion, user can use his voice. Introduced system rec-ognizes couple of basic keywords: “pause”/“resume”,

Figure 4. Widgets of projected ARTable GUI extended by visualization mode. Left: List of programs, wheregreen programs are ready to visualize or run and red programs need to set parameters. Middle: List ofinstructions of currently visualized program. Visualization is fully controllable with provided buttons –“Pause”/“Resume”, “Stop”, “Replay”. Visualization can be also controlled with voice by saying same keywordsas are names of those buttons. Right: Rectangles defining areas on the table onto which the grabbed objects ofdifferent type will be placed. Dashed polygon defines area in which the robot will apply glue to correspondingobjects. All these supplementary elements of projected GUI are displayed at the same time when programvisualization runs.

(a) Virtual robot gripper is grabbingvirtual object from the feeder.

(b) The robot gripper moves grabbedobject to specified place pose.

(c) The robot is placing virtual objectinto preset position.

(d) The robot goes for smaller virtualobjects – stretchers.

(e) Simulation of glue application tovirtual objects within defined area.

(f) Result of “Stool assembly” programrunning on the real robot.

Figure 5. An example of program visualization. In this case – first block of “Stool assembly” program whichconsists of a total of three blocks. In this block the robot will pick the objects from the feeder, place them ontothe table and apply glue into the holes in order to prepare these components for user to assembly. This blockconsists of eleven instructions – “wait until user finishes”, four “pick from feeder” and consequent “place topose” instructions, “apply glue” and lastly “get ready”.

“stop”, “replay”, “next” (to immediately move to nextinstruction) and “back” (to immediately move to pre-vious instruction). These keywords are insured againstunexpected behavior. For example if users says “stop”when visualization is already stopped, he gets notifiedby speech synthesizer.

During whole visualization process, the HoloLensand the ARTable keeps communicating using afore-mentioned rosbridge. Both systems synchronizes theirstates about which instruction is visualizing, whetherit’s playing or is stopped and so forth. This continuouscommunication ensures scrolling down in instructionlist of projected 2D interface or enabling and disablingprojected buttons. Lastly, supplementary elements ofprojected interface – such as object place pose rectan-gles, polygons defining areas from which to pick or inwhich to apply glue to objects – are displayed at onceto help the user get an instant overview of an outcomeof current program.

6. ConclusionsIn this paper, usage of mixed reality head-mounteddisplay was integrated into a human-robot collabora-tive workspace – the ARTable. Current 2D projectedinterface was extended over the visualization mode,which adds new buttons and functionality in order tosuccessfully communicate with the headset. A visu-alization system that enhances user understanding ofrobotic programs in the ARTable workspace was im-plemented. This system enables the user to visualizelearned programs by a form of animation, allows himto control this animation by his voice or buttons ofthe projected interface and guides him by speech syn-thesizer through whole visualization process. Withpossibility of controlling the visualization, even anexperienced user might find it usable, when he canquickly see program’s outcome without the necessityof running the real robot. Use of the headset allows tovisualize valuable spatial information, such as robotperception, which enhances human-robot collabora-tion.

Provided solution enables to use multiple head-sets, where all of them are synchronized through theARTable. Thanks to that, they use same informationand render same holograms. Solution of the ARTable’spart is headset independent.

For future plan, this work will be extended overa functionality, that will interactively guide a userthrough programming of the ARTable programs withuse of speech synthesizer and 3D holograms. Furtheron, proper user testing will take place.

AcknowledgementsI would like to thank my supervisor Michal Kapinusfor his help.

References[1] Zdenek Materna, Michal Kapinus, Vıtezslav Be-

ran, and Pavel Smrz. Using Persona, Scenario,and Use Case to Develop a Human-Robot Aug-mented Reality Collaborative Workspace. In HRI2017, pages 1–2. Association for Computing Ma-chinery, 2017.

[2] Justin Huang and Maya Cakmak. Code3: A Sys-tem for End-to-End Programming of Mobile Ma-nipulator Robots for Novices and Experts. In Pro-ceedings of the 2017 ACM/IEEE InternationalConference on Human-Robot Interaction, HRI’17, pages 453–462, New York, NY, USA, 2017.ACM.

[3] Susanne Stadler, Kevin Kain, Manuel Giuliani,Nicole Mirnig, Gerald Stollnberger, and Man-fred Tscheligi. Augmented Reality for Indus-trial Robot Programmers: Workload Analysis forTask-based, Augmented Reality-supported RobotControl. In 2016 25th IEEE International Sympo-sium on Robot and Human Interactive Communi-cation (RO-MAN), pages 179–184, Aug 2016.

[4] Jan Guhl, Son Tung, and Jorg Kruger. Con-cept and Architecture for Programming IndustrialRobots using Augmented Reality with MobileDevices like Microsoft HoloLens. In 2017 22ndIEEE International Conference on EmergingTechnologies and Factory Automation (ETFA),pages 1–4, Sept 2017.

[5] Ivo Maly, David Sedlacek, and Paulo Leitao.Augmented Reality Experiments with IndustrialRobot in Industry 4.0 Environment. In 2016IEEE 14th International Conference on Indus-trial Informatics (INDIN), pages 176–181, July2016.

[6] Eric Rosen, David Whitney, Elizabeth Phillips,Gary Chien, James Tompkin, George Konidaris,and Stefanie Tellex. Communicating Robot ArmMotion Intent Through Mixed Reality Head-mounted Displays. CoRR, abs/1708.03655, 2017.

[7] Ivo Sluganovic, Matej Serbec, Ante Derek, andIvan Martinovic. HoloPair: Securing SharedAugmented Reality Using Microsoft HoloLens.In ACSAC, pages 250–261, December 2017.

[8] Bonnie Christian. HoloLens trial gives doctors’X-ray vision’ to allow them to peer inside

patients during surgery. WIRED, March 2017.http://www.wired.co.uk/article/industries-using-microsoft-hololens.

[9] Laura Toler. Holographic Rovers: AugmentedReality and the Microsoft HoloLens. Technicalreport, NASA Kennedy Space Center; CocoaBeach, FL United States, April 2017.

[10] Oliver Kreylos. HoloLens and Field of Viewin Augmented Reality, August 2015. http://doc-ok.org/?p=1274.