Post on 09-Nov-2018
STO-MSG-111 09 - 1
Augmented Reality and Mixed Reality Technologies:
Enhancing Training and Mission Preparation with Simulations
Joni A. Amorim, Dr. University of Skövde - HiS
Kanikegränd 3, Box 408, SE-541 28, Skövde
SWEDEN
joni.amorim@gmail.com
Carlos Matos, 1º Ten Instruction Center for Operations on Law and Order Assurance - CIOpGLO, Exército Brasileiro
28º Batalhão de Infantaria Leve, Av. Soldado Passarinho s/nº, Campinas
BRAZIL
eb.carlosmatos@gmail.com
Ana R. M. Cuperschmid, M.Sc. School of Civil Engineering, Architecture and Urban Design
University of Campinas - UNICAMP, Av. Albert Einstein 951, Campinas
BRAZIL
fale@anacuper.com
Per M. Gustavsson, Dr. Swedish National Defence College - Försvarshögskolan
Drottning Kristinas väg 37, 115 93, Stockholm
SWEDEN
per.m.gustavsson@fhs.se
Cesar T. Pozzer, Dr. Universidade Federal de Santa Maria, Centro de Tecnologia (CT) - Prédio 07
Dep. Eletrônica e Computação, Campus Universitário, 97105-900, Santa Maria
BRAZIL
pozzer@inf.ufsm.br
ABSTRACT
The Instruction Centre for Operations on Law and Order Assurance (CIOpGLO) is a Brazilian Army facility
created in March 2005 in Campinas, Brazil. The mission of this centre involves the offering of training in
different perspectives, which includes the preparation of soldiers to enter in slums areas in Rio de Janeiro
and other cities to arrest criminals, whenever a federal intervention is required. This centre is involved in
training to guarantee law and order and, at the same time, prepare officers and soldiers for interventions
even in urban areas. To allow such training, this facility counts with physical built sites to allow soldiers to
train how to get inside houses, how to shoot at short ranges (from 0 to 30 meters), how to move and shelter
while going up in a hill with many houses and corridors on the way, and so on. The Brazilian Army, in the
last few years, started operating in slums like the ones of the "Alemão" and the "Penha" complex in Rio de
Janeiro. The Army is also participating in operations out of Brazil in countries like Haiti. In situations like
Augmented Reality and Mixed Reality Technologies
09 - 2 STO-MSG-111
this, the armed forces take over the coordination of public security temporarily to recover the control of
certain areas. Since the armed forces were not originally created to act in situations like this, there is a need
to train all military stakeholders involved so that the operations are successful. Additionally, major events
like the Confederations Cup, the World Cup in 2014 and the Olympics in 2016, generate additional demands
for the armed forces, which are likely to be called to act at specific times. Moreover, it is noteworthy that
there is a growing trend in which conflicts around the world occur, more than ever, inside the cities, where
civilians take great risk and suffer many casualties, something called as “collateral damage” of the urban
warfare. Recent examples include Afghanistan and Iraq. In this work, the preparation of soldiers at
CIOpGLO is discussed while the possibility of using new approaches based on augmented reality and mixed
reality technologies are considered. As a way to enhance training and mission preparation with simulations,
this research focus on augmented reality (AR) supported by head-mounted displays (HMDs). HDMs may
have many shapes, which include pairs of glasses with lenses that present AR with superposed images,
enabling its wearer a total immersion in the simulation. The method used in this work involves a literature
review on AR and HMDs, assessment of training needs at the Brazilian Army and an evaluation of emerging
technologies from the ICT sector. The technologies to be considered are the HDMs, in this specific case the
available programming languages, software and hardware from suppliers of commercial off-the-shelf
(COTS) and military off-the-shelf (MOTS). The main contribution of this work is the comparative study of
the main solutions for HDM. This study represents an essential step for concept development and for the
experimentation to exploit and evaluate the use of simulations. The research presented suggests that the
approach is effective and that future work should be on both development of new applications and its
evaluation in real training sets in Brazil.
1.0 INTRODUCTION
The Brazilian Army, in the last few years, started law enforcement operations in slums like the "Alemão"
and the "Penha" complexes at Rio de Janeiro. The Army took part on United Nations (UN) peacekeeping
operations out of Brazil, such as MINUSTAH, at Haiti, and UNMIT, at East Timor. In situations like this,
the armed forces take temporary control of public security, in order to restore order to certain critical areas.
Since the armed forces were not originally created to act in situations like this, it is imperative to train all
military staff that is going to be deployed, for the success of these operations [1]. Additionally, major events
like the Confederations Cup, the World Cup in 2014 and the Olympics in 2016, generate additional demands
for the armed forces, which are likely to be called to act at specific times. Moreover, it is noteworthy that
there is a growing trend in which conflicts around the world occur, more than ever, inside the cities, where
civilians take great risk and suffer many casualties, something called as “collateral damage” of the urban
warfare. Recent examples include Afghanistan and Iraq.
The Instruction Centre for Operations on Law and Order Assurance (CIOpGLO) is a Brazilian Army facility
created in March 2005 at Campinas, Brazil. This centre main objectives are to develop Techniques, Tactics
and Procedures (TTP) for law enforcement operations and to train military personnel on both law
enforcement and urban operations. The students that apply for the courses offered by CIOpGLO come from
many ranks, from sergeants to captains, and the courses offered each year vary from 1 to 4 weeks long. At
the end of these courses, soldiers are ready to enter urban hostile, areas like the slums in Rio de Janeiro, to
arrest criminals, whenever a federal intervention is required [2].
CIOpGLO facilities include a modular simulated slum, a tactical entry modular house, a container-based
simulated neighbourhood and a rappelling tower, with doors and windows for alternative entry methods.
These are all very important on teaching instinctive shooting (from 0 to 30 metres), how to enter buildings
with armed enemies and how to move and shelter along narrow alleys, uphill and multi-ways intersections.
Almost all trainings are force-on-force (the simulated hostiles shoot back), using paintball markers, non-
lethal weapons, explosives and personnel representing unarmed civilians to create a realistic urban
Augmented Reality and Mixed Reality Technologies
STO-MSG-111 09 - 3
operations environment.
Figure 1: Military force-on-force training in a simulated slum at CIOpGLO.
Figure 2: Masks in use during trainings at CIOpGLO in Campinas city, Brazil.
Augmented Reality and Mixed Reality Technologies
09 - 4 STO-MSG-111
Figure 3: Instructors supervise soldiers during training at CIOpGLO in Campinas city, Brazil.
Figure 4: Training at CIOpGLO in Campinas city, Brazil.
Augmented Reality and Mixed Reality Technologies
STO-MSG-111 09 - 5
Figure 5: Brazilian Army training at CIOpGLO in Campinas city, Brazil.
As a consequence of its growing importance, the project of a new urban warfare training compound for
CIOpGLO is on its way: a multi-blocks simulated city with all the infrastructure of a real city (commercial
area, houses, bus station, school, and more). The objective of these facilities is to offer the students a more
realistic training. And realism is essential: a most important statement on military simulations relies on the
induction of combat stress. With the use of paintballs, mock explosives and a gradual exposure to stress, the
instructors of the CIOpGLO are able to see their students showing the same symptoms of combat stress –
tunnel vision, temporary loss of recent memories, etc. [3]. These reactions during simulations reveal how
these soldiers would act like on real situations – therefore, stress control can be trained and those who do not
have emotional control are usually put away from combat missions.
It is interesting to point out that in simulated cities like this, the possibility of using new approaches based on
augmented reality and mixed reality technologies should be considered. New opportunities for military
simulation and training are emerging by the use of Mobile Augmented Reality (AR). In conventional Virtual
Reality, users immerse in a complete synthetic world, while in mobile AR they interact with a real world
environment and in the same time the user receives additional virtual visual data. By merging a range of
digital and physical media, AR can be enriched by different perceptions and comprehensions offered by both
physical and virtual environments. Mobile devices are usually free of cords and not tied to a static space for
interaction, so that when used by military training the process becomes borderless.
As a way to enhance training and mission preparation with simulations, this research focus on augmented
reality (AR) supported by head-mounted displays (HMDs). We do not include Virtual Reality (VR) devices
in this research. For a more realistic training, AR devices have the advantage over VR devices because they
can use the real environment information with superposed virtual information. This allows the user to easily
Augmented Reality and Mixed Reality Technologies
09 - 6 STO-MSG-111
move while interacting with real surroundings. Pure VR environments present serious drawbacks to allow
the user to move freely, and although some devices such as Virtuix Omni [4] can recognize some body
motion, it is limited to legs and short torso movements.
The method used in this work involves a literature review on AR and HMDs, assessment of training needs at
the Brazilian Army and an evaluation of emerging technologies from the ICT sector. The technologies to be
considered are the HDMs, in this specific case the available programming languages, software and hardware
from suppliers of commercial off-the-shelf (COTS) and military off-the-shelf (MOTS). The main
contribution of this work is the comparative study of the main solutions for HDM. This study represents an
essential step for concept development and for the experimentation to exploit and evaluate the use of AR on
military simulations.
2.0 MOBILE AR AND THE BUILT ENVIRONMENT
In this paper, we discuss the development of a new training system using AR technology, for military
simulations. To make this possible it is necessary to study the interaction of virtual elements with the real
environment, so the use of AR in the field of Architecture, Engineering and Construction (AEC) is
investigated. Mobile devices are being experienced in academic research and commercial uses in AEC. For
example, Irizarry et al. [5] developed a mobile AR tool for Facility Managers to access information about the
facilities by providing a tool for visualizing the real environment with added interactive data. The civil
engineering firm Bechtel uses the applications Junaio and Autodesk Mobile 360 to superimpose virtual
diagrams of mechanical, electrical and plumbing on the current images of the construction [6]. The German
company, Pure Gruppe Architects, performs an overlap of 3D model in the exact position where the future
building will be constructed [7]. The Netherlands Architecture Institute displays in AR the Municipal Market
of Rotterdam in the place where the construction of the building is being carried out [8]. Allen et al. [9]
developed a prototype smartphone AR system as a tool for aiding public participation in urban planning. It
superimposes virtual 3D models over an existing building and allows users to provide feedback based on
their personal preference of the proposed designs. Shen & Jiang [10] proposes a mobile AR application for
Communication, Collaboration, and Learning (CCL) of building MEP Systems. In order to understand how
Building Information Modeling (BIM) models could be used to create an environment for mobile 3D game
for serious purpose, a BIM model of an institutional building to be used in a game application for iPad was
developed.
In all those cases a common concern is precise positioning, scalability, and wireless connection. The
positioning depends on the registration method adopted. According to Yabuki et al. [11] there are three
methods: 1) marker-based - uses the standard marker-tracking technologies to superimpose virtual objects; 2)
GPS, gyroscopes, accelerometers and a compass; 3) feature points on the video image – utilized by
markerless technologies that detect exclusive features of environments to establish where to superimpose
virtual objects. In complex environments, such as constructions fields, the markerless tracking is more
appropriate.
Smartphones and tablets can use geographic referencing through GPS technology, positioning via WLAN,
and Cell-ID to determine the position [12]. In general, AR apps use GPS to establish the location of points of
interest and 3D virtual models. GPS is the most precise and allows for an accuracy location around 10 meters
but requires several seconds to minutes to determine the position [13]. Inside the buildings this technology is
even less accurate, with a deviation up to 500 meters [13]. Aware of that, Wang et al. [14] proposes an AR
system operating in conjunction with tracking and sensing technologies such as radio frequency
identification (RFID), laser pointing, sensors and motion tracking. Chi et al. [15] presents other outdoor
localization technologies such as ultra-wide band (UWB) and barcoding that can be used in conjunction with
sensors to provide the exact geometric information for AR extracting from real world information captured
with cameras and laser range finders. In order to precisely locate building models on urban environment,
Yabuki et al. [11] proposed a new registration technique using point cloud and natural feature points. In this
Augmented Reality and Mixed Reality Technologies
STO-MSG-111 09 - 7
approach it is possible to link several feature points of video display and corresponding points of the point
cloud. The system uses the principle of photogrammetry to accurately make the registration of the points.
With this technique it is possible to select natural feature points and points for error checking by comparing it
with the original building model.
An important factor to be observed is the definition of the space in which the virtual and real objects will
coexist. It is necessary to establish spatial and contextual links with each other and proper registration 3D
objects inside the augmented scene. Therefore, it is essential to determine the proper relationship between
virtual models, the location and dimensions of the real world to ensure the appropriate combination of the
two worlds, creating the illusion that the two coexist in a single scene and work the same way [10]. In order
for that to happen it is necessary not only the right positioning of the model, but the correct scalability, and
fast display. To gather a 3D virtual model to display immediately it is important to consider the possibility of
preloading all the models needed in the application before experiencing AR.
As exposed in this section, there is lot of research concerning the use of mobile AR and the built
environment. Each one takes one approach of tracking, display and interaction method. Analysing the
strategies, individually, is possible to detect its pros and cons; there is no perfect method. Sometimes a
custom development is needed in order to satisfy certain requirements. In order to evaluate HMDs to military
training, first it is necessary to observe how it is done nowadays and in what manner this technology could
be helpful.
3.0 MILITARY TRAINING NEEDS AND AR
In order to evaluate HMDs to military training, first it is necessary to observe how it is done nowadays and in
what manner this technology could be helpful. The best COTS virtual shooting simulators available use large
projection screens, may have a high level of precision, graphics and Artificial Intelligence (AI), but lack the
need of the user to move himself, to use cover correctly, or to interact with the environment [16]. They still
feel as a video game, especially because the user can still see a flat screen and distinguish it from the real
floor, real roof and surrounding real objects or companions – in case of multiuser simulation
[17][18][19][20]. But what if soldiers could use real-life scenarios, having to move fast, dash into cover, and
avoid fire shot by virtual enemies, who interact with the same real objects he sees? The incorporation of
virtuality into the real world would push the user, during the training exercise, to the same behaviour and
movements as would happen in real dangerous situations.
As one of the most important parts of Military Simulations is the immersion (so trainees can be inducted to
combat stress, in order to develop self-control), there must be a special attention on the manipulation of the
subject’s fours senses: vision, audition, olfaction and tact. The main one is the vision, responsible for 75% of
the information entering the brain [21], usually well explored by most virtual simulators, but “sound is
integral to total-immersive experience”. It is important to remember that, in order to absorb the experience, it
is “indispensable harmony with the other senses” [21] of the user. If well combined, these can make a
simulation extremely useful for foot soldiers. The incorporation of virtuality into the real world would assure
to the user, during the training exercise, the same behaviour and movements as in real dangerous situations.
This level of simulation could become reality with a sharp combination of some technologies today´s top
computer engineering companies are developing:
A high-definition AR device would have its user see the real world with virtual additions, such as
3D Computer-generated imagery (CGI) enemies, vehicles, civilians and more. Explosions, shootouts
and many other situations could be virtually simulated within the real scenario. This AR viewing
devices are available as HMDs or AR glasses, like the zSight [22] or the Google Glass [23], or as
contact lenses as iOptik [24].
Augmented Reality and Mixed Reality Technologies
09 - 8 STO-MSG-111
3D mapping of the specific location where the simulation would take place (using geographically
referenced information), could allow the interaction between the virtually generated objects and the
real environment, creating an outstanding level of realism. Therefore, computer-generated elements
could use the actual terrain to perform their tactics, just as real enemy would. This was done by the
US Army Battle Lab, at Fort Benning [25].
Additionally, realistic sound effects would make the user react to virtual events - from an explosive
device triggering, to a supporting airstrike -, improving immersion. Although, this is not achieved
just by having a surround sound output. The software's spatial sound must calculate “the correct
volume, filtering and reflections of the sound or reverberation of objects at distance”, in a way that
“the feeling of presence during gameplay is often enhanced by the use of spatial sound. It should
offer “sensory ‘proof’ and convinces the [user] of the virtual world” [26].
Interaction with virtual elements without the need to touch a screen or push a button could be
accomplished using special devices, such as gloves with sensors [27] or simply shooting at virtual
AR with laser transmitters, installed on real weapons [28]. Virtual enemies could shoot back and be
able to make the user suffer consequences for been hit, as already occurs in an iPhone/iPad game
that uses this type of AR interaction: it is the AR Hunter, developed to the Parrot Drone [29]. To
simulate the injuries, a special vest could “harm” the user with pressure points [30], small electrical
discharges [31][32] or compressed air [33], which are features already offered COTS products.
All of these innovative approaches could grant military training an extremely realistic simulation, one that
would be an alternative to the live ones, which have high costs on acting personnel, training ammunition,
pyrotechnics and other special effects. Besides, this kind of simulation would necessarily be more physically
demanding and more immersive than a simple point-and-shoot virtual simulation, for it would have the user
move himself considerably more than in front of a screen or inside a hexagon or a dome – like the system
developed by Raytheon and Motion Reality [34][35]. Additionally, such mixed environment (real + virtual)
can reduce life risks since there is no need to use real ammunition and can also reduce costs, in the same way
as using virtual vehicle simulators.
4.0 COMPARATIVE STUDY OF HMDS
For the military training, the use of head attached displays in AR systems allows great freedom of movement
and provides the participants an individualized and direct viewing around their eyes. This way, each user has
both hands free for any interaction needed with the real world. Head attached displays require the user to
wear a visualization system. There are three different types, depending on the technology: HMDs, retina
devices, and head mounted projectors.
According to Bimber and Raskar [36], HMDs are the display devices more used for applications that use
some type of head attached display. The HMDs are part of a class of devices used in immersive AR,
characterized by the ability to see directly the world around the observer through the media, reaching the
maximum possibility of presence and level of imagery quality. Milgram et al. [37] draw attention to the fact
that such optical devices require accurate tracking and low latency of the movements of the user's head.
Moreover it is important to observe the accuracy of the calibration point of view and the suitability of the
field of vision, in addition to the discomfort while using the device (it can cause headaches and eyes fatigue)
[38].
There are two different HMD technologies: optical-based and video-based [36][39][40]. Optical-based
systems typically use devices formed by lenses that allow the user to see the real world with the virtual world
designed on the lenses, which are positioned in front of the eyes. The translucent lenses allow the user to
directly look at it to see the real world. Video-based systems use video cameras that provide for the user a
view of the real world. These cameras' videos are combined with virtual images, generating a scene mixed
Augmented Reality and Mixed Reality Technologies
STO-MSG-111 09 - 9
with both the real and the virtual worlds. The result can be visualized by the HMD that provides a vision
narrowed by having a display in front of the user's eyes. According to Livingston et al. [41] optical-based
devices have the advantage of allowing the user to have the peripheral vision around the display and natural
vision of the real world at the same time. On the other hand, video-based systems have the advantage of
more control over the occlusions between real and virtual entities, but the user vision is limited to the
geometric and colour resolution of the camera that captures the real world.
All head attached displays require the user to use the visualization system he/she is wearing. Schmorrow et
al. [40] and Bimber and Raskar [36] present an exotic form of head attached display, a scanning device that
uses laser retinal low power to design lights directly onto the retina of the user. Among the head attached
displays, there are the head-mounted projector devices, which make use of projectors in miniature, or LCD
panels in miniature, with backlight and images projected on the surface of the real environment.
There are several HMDs for AR commercially available; however, its use in military training is still
restricted. Aiming to modify this panorama and increase the possibilities of the use of these devices, a
collection and classification of their main characteristics is performed. There are some aspects of HMDs that
should be compared in order to have a good experience with AR. For a selection of which features to be
evaluated, there are some technical considerations:
Input Method: SDK (Software Development Kit) / API (Application Programming Interface).
Angle of view: The angle of view has to be observed. If it were too narrow, peripheral view might
be out of the screen, which would lower immersion sensation. Livingston et al. [41] point out that
HMD displays reduce human visual capabilities in several ways.
Resolution: The higher the resolution, the better. The intention is offer a realistic mixed visualization
of the two worlds, real and virtual.
Processing power to run live AR: The response time has to be taken into consideration, once delay is
an annoying aspect that could discourage the use AR- this could be also caused by a connection
issue. The proposed AR package is expected to feature high quality multimedia content that could
take a while to display so the latency should be observed.
Sound: Surround sound system attached to HMD, so the user could have total-immersive experience
and also a headset to allow the users to communicate with each other.
Battery life: Most live simulations at CIOpGLO are at least 30min long. Besides, at the instruction
centre they usually have up to 3 groups of military students/trainees that alternate on a rotation
instruction method. This way it is desirable a battery life of 90 min with full power.
Mobility: The mobility of the system is essential for the AR military training [41]. The users must be
able to run, dash and prone. It should be a wireless HMD, or with a reduced number of hardened
wires.
Tracking system: A robust and accurate tracking system would be very important [41]. The
hardware must have at least a GPS tracking system in order to provide information about the
surrounding environment through geographic positioning of the virtual opponents and the allied
units, and also to allow post-action evaluation of the performance and actions of the users.
Additionally, real-time position monitoring, combined with wireless communications, can allow the
instructor to observe actions and feed automated generated forces to react in accordance to the
position and actions of the users.
Augmented Reality and Mixed Reality Technologies
09 - 10 STO-MSG-111
Weight: Weight is another important aspect, once the user has to wear this and act like it would in
battlefield [41].
Size: Adequate size to feel comfortable [41].
Price: It's always a concern, therefore, each estimated value is shown.
Major consumer electronics firms have developed and introduced recent HMD technology, as the case of
Google, Epson, Vuzix, Laster, Epiphany, Optinvent, Recon Instruments, Innovega, GlassUp, BrilliantService
and Telepathy. Based on the stage of development of the commercially available devices, Table 1 shows a
comparison of some HMDs. This comparison could be used to choose one of them to be tested on the
proposed military training.
Table 1: Comparative study of HMDs commercially available.
Glass1 Moverio BT-1002 Smart Glasses M1003
META.014 Laster MG15
Developer Google Epson Vuzix Meta Laster Technologies
Input Method Glass Development Kit (GDK)
Moverio SDK Vuzix M100 SDK Meta Unity3D SDK Multi-layer SDK
Information not available
Operating System Android 4.0.3 and higher
Android 2.2 and higher
Android Meta OS Information not available
Field of view Prism screen is above the right eye
23 degrees 16 degrees - right eye
23 degrees 40 degrees
Display 640×360 pixels 960x540 pixels 400x240 pixels 960x540 pixels 800x600 pixels
Image/Video - Capture
Photos 5 MP, HD 720p
Photos 5 MP, HD 720p
HD 720p
HD 720p
Information not available
Audio Information not available
Information not available
Information not available
Information not available
Information not available
CPU OMAP 4430 SoC, dual-core
Information not available
OMAP4430 at 1GHz
Information not available
Information not available
Memory 1GB RAM 1GB RAM 1GB RAM Information not available
Information not available
Battery 24 hours for typical use
About 5.8 hours Up to 8 hours hands free 1 hour hands free+display +camera
Information not available
Information not available
Connectivity Wifi 802.11b/g/n Bluetooth
Wifi 802.11b/g/n USB: microUSB
Wifi 802.11b/g/n Bluetooth USB
Information not available
Information not available
Tracking GPS + 9 axis head tracking
9 axis head tracking
GPS + 9 axis head tracking
9 degree of freedom sensor via USB + Surface Tracking
Object-Recognition (Add-on)
Storage 16 GB Flash total (12 GB of usable memory)
microSDHC (32GB maximum)
4GB flash + external microSD (8GB maximum)
Information not available
Information not available
Weight 50g 240g glass 165g controller
Information not available
Information not available
Information not available
Augmented Reality and Mixed Reality Technologies
STO-MSG-111 09 - 11
Glass1 Moverio BT-1002 Smart Glasses M1003
META.014 Laster MG15
Price Explorer version: $1500 USD Consumer Edition: $300-500
$699,99 USD Information not available
$667,00 (April 2014)
Information not available
More information It might be harder to see the Glass screen in bright sunlight.
Bright display designed for indoor and outdoor use.
320x240 Infra-Red Depth Camera via USB
It can be used for outdoor applications where the environment is very bright. Currently not accepting 3D models.
1. https://support.google.com/glass/answer/3064128?hl=en 2. http://www.epson.com/cgi-bin/Store/jsp/Moverio/Home.do?ref=van_moverio_2012-03-001 3. http://www.vuzix.com/consumer/products_m100.html 4. http://www.meta-view.com 5. http://laster.fr/produits/MG1/
For live military training, the main features that should be analysed are the ability to superpose virtual
images and the field of view. According to the human field of view, those devices may be categorized into
two groups: reduced and unchanged field of view. Because Google Glass and Smart Glasses M100 use only
a small screen to project the AR content, they don’t change the user’s perception about the environment,
even the peripheral vision. All the others (Moverio BT100, Meta.01 and Laster MG1) are very similar, so
that the field of view becomes much reduced. In practical situations, the limited field of view may reduce the
perception about the real scenario, thus negatively interfering on the performance and fairness of the
simulation.
Google Glass and Smart Glasses M100, instead of the other AR devices, do not project the virtual image
over the whole retina (glass’s field of view). Only a small portion of the viewing (upper right corner) can be
enhanced. In military context, it seems only appropriate to show short information, like routes and basic data.
It cannot handle, for example, the exhibition of a moving virtual enemy superposed to the real scenario.
Additionally, Glasses M100 doesn’t use a translucent screen, so the field of view is partially reduced.
Moverio BT100, Meta.01 and Laster MG1 use two translucent screens to project the augmented content onto
the surface of the real environment. This feature allows the exhibition, for example, of virtual animated
enemies overlaid to the real scenario. Laster MG1 is the device with the largest field of view (40 degrees),
but not yet comparable to the VR Oculus Rift, which covers >90 degrees horizontally [42].
Another technical issue that must be taken into consideration is the occlusion - the effect of one object in a
3D space blocking another object from view. The development of 3D occlusion algorithms to mesh the real
and virtual world together in real time is demanding a lot of research and still can be considered a work in
progress [43].
Additionally to the evaluated features, the HMDs must be stable and capable of withstanding the instabilities
of the environment, so the resistance to dust, impacts and water should be considered. These could not be
evaluated since this study is based on the information of the devices provided by the developers.
Besides the technical issues, Livingston et al. [41] call attention for designing a user interface that do not
distract the user’s attention from the task. Ideally, the physical actions needed to execute functions in the
Augmented Reality and Mixed Reality Technologies
09 - 12 STO-MSG-111
interface and the ease of understanding what is presented must be as simple and intuitive as possible. The use
of metaphors, voice commands and gestures that military personnel currently use are essential to build the
user interface. Test must be conducted in order to ascertain the user acceptance.
5.0 FUTURE WORK
Future work will include the development of simulators and serious games as a way to investigate the
potential of gamefied training.
The investigation on the use of mobile AR applications for gamefied military training is both complex and
interdisciplinary. In order to deal with the complexity, this research project was divided in six phases to be
executed by a set of professionals from different disciplines. In the next paragraphs, an overview of the
gamification management is presented while using the terminology and the best practices suggested by [44].
It is important to notice that [44] discusses two prevailing methods for developing gamification efforts, the
ADDIE process and the Scrum approach, and suggests the use of a hybrid of the two models that should be
modified accordingly to each project: determine outcome of the learning, determine the type of content to be
taught, develop a rough storyline, create the gamification design document, create a paper mock-up of the
game and play it, create storyboards and concept art, test the storyboards and concept art by showing it to
focus groups, have play-tests and daily meeting during the development, and so on. It should be highlighted
that, from the six phases to be presented in the next paragraphs, phase 5 would be the one to use the hybrid
approach.
Phase 1 refers to the understanding of the simulation and game elements that are relevant to this new mobile
AR application. Ideally, the set of professionals would include at least the following: a specialist on AR, one
or more subject matter experts and a designer of multimedia based learning solutions with knowledge on
simulation and game development.
Phase 2 refers to the preliminary use of a head-mounted display (HMD) in a helmet or in a pair of glasses
that would present information from a GPS device and that would record videos during the training. The
GPS device could be the one from a mobile phone and would allow instructors to better understand how the
students move inside of the simulated city. In this phase, the students would still be using paint ball weapons
and the HMD would not superpose images of characters like civilians and criminals. Ideally, the set of
professionals would include the ones mentioned for phase 1 and a project manager to coordinate the
activities and provide detailed planning using both traditional and agile methods. The team would benefit
from including a representative of the learner population in the ideation phase and for the preliminary tests.
Phase 3 refers to the superposition of images of characters like civilians and criminals at the HMD. Since the
training must happen both during the day and the night, the presentation of images on the HMD should rely
on GPS information and/or on sensors spread in the training facility. The images of characters would have
movement to increase the level of realism while the earphones from the HMD and/or the speakers from the
training facility would synchronize the appropriate sound effects. Despite the fact that the superposition of
images may happen using only the HMD associated with a mobile device like a smartphone, the best
solution would include the development of a software system that would automate the caption of images and
videos from the HMD and that would control the presentation of sound and images to the students. The
proposed software system would interface with a Learning Management System (LMS) in order to better
administer, document, track, report and deliver the training since it may also include the use of
complementary material for self-study like hypertexts, videos and so on. The replica weapons would still be
the ones based on paint ball technology. Ideally, the set of professionals would include the ones mentioned
for the previous phases and others like a programmer to develop the software system, an information
technology representative to ensure that the new software system runs properly on the organization’s
computer infrastructure and a specialist on the specific LMS in use. The team would also benefit from
Augmented Reality and Mixed Reality Technologies
STO-MSG-111 09 - 13
having an animator to develop the characters and a sound technician to be responsible for how the game
sounds.
Phase 4 refers to the improvement of the replica weapons so that the software system would be able to
associate their use to the many events, which includes the storage of data related to aiming and firing the
weapons. This improvement would allow the instructors to better understand how the students performed.
Ideally, the set of professionals would include a specialist on replica weapons used for military training with
knowledge on AR.
Phase 5 refers to the gamification. While keeping the instructional objective needs as the main priorities, this
phase will focus on designing and developing a game that is engaging, that aids in retention and that is
impactful. The game would benefit from the infrastructure developed in the previous phases and on the
knowledge derived from using mobile AR on this specific case of military training, a necessary approach to
avoid the trivialization of learning so that the gamefied learning would be challenging and better than
traditional simulations. Ideally, the set of professionals would include an instructional game designer to link
the instructional objectives with the game play, an artist to create the “look and feel” of the game and a level
designer to create different levels of difficulty based on different challenges and parameters.
Phase 6 refers to the incorporation of new peripherals that would increase the level of realism of the training.
In this way, the research would go beyond human augmentation with HMD devices using AR to include
bioacoustics sensing, computer-brain interfacing, biometric sensing, control by speech, speech recognition,
natural language processing, touching, haptic, gesture recognition, behavioural analytics, eye tracking,
wearable computer devices, affective computing, mood and emotion recognition, and so on [45]. Ideally, the
set of professionals would include a human computer interface specialist for specific considerations related
to ergonomics and usability in a context of multimodality and mobility.
6.0 CONCLUSION
The use of AR in military training requires a high level of automation and integration of information with
physical resources. However, the effective integration of virtual information (3D models and animations)
with the physical built site is a challenging proposition that has to be overcome to achieve immersive
sensations. The tracking/sensing for context aware is considered to be crucial for enabling visualization, but
also for dynamic interaction with the training activity. Also, a sophisticated control method has to be
developed with the addition of novel sensors to provide a user interface that considers natural human
behaviours, such as in Project Glass [23]. The technology must be part of the simulation in a natural way, so
it would not act as a barrier for a real training. It is already possible to conclude that military training may
benefit from the incorporation of AR. Future work will involve implementing the phases described in the
previous section.
ACKNOWLEDGMENT
The authors would like to thank the following organizations for their support during the development of this
work: UNICAMP (http://www.unicamp.br/), USP (http://www.usp.br/), Exército Brasileiro
(http://www.exercito.gov.br/web/guest), SAAB AB (http://www.saabgroup.com/), CISB (http://cisb.org.br/),
CNPq (http://www.cnpq.br/) and FAPESP (http://www.fapesp.br/).
Augmented Reality and Mixed Reality Technologies
09 - 14 STO-MSG-111
REFERENCES
[1] Gomide, R. (2012). Exército treina para Garantia da Lei e da Ordem e ‘guerra no meio do povo’. iG.
Retrieved August 19, 2013 from http://ultimosegundo.ig.com.br/brasil/2012-08-27/exercito-treina-para-
garantia-da-lei-e-da-ordem-e-guerra-no-meio-do-povo.html
[2] COTER (2013). CI Op GLO. Retrieved August 19, 2013 from
http://www.coter.eb.mil.br/index.php/acervo/centros-de-instrucao/83-centro-de-instrucoes/118-ci-op-glo
[3] CIOpGLO (2012). Classified Report: National Short-Course on Urban Warfare 1/2012. Brazilian
Army. Campinas, Brazil.
[4] Virtuix Omni (2013) . Retrieved August 19, 2013 from http://www.virtuix.com
[5] Irizarry, J., et al. (2012). InfoSPOT: A mobile Augmented Reality method for accessing building
information through a situation awareness approach. Automation in Construction. Retrieved August 19, 2013
from http://dx.doi.org/10.1016/j.autcon.2012.09.002
[6] Apple (2013). iPad in business. Retrieved August 19, 2013 from
http://www.apple.com/ipad/business/profiles/bechtel/
[7] Junaio Blog (2010). Augmented Reality in urban context. Retrieved August 19, 2013 from
http://junaio.wordpress.com/2010/07/24/augmented-reality-in-urban-context/
[8] Netherlands Architecture Institute (2009). See what is not (yet) there – with the NAI and Augmented
Reality. Retrieved August 19, 2013 from http://en.nai.nl/museum/architecture_app/item/_pid/kolom2-
1/_rp_kolom2-1_elementId/1_601695
[9] Allen, M., Regenbrecht, H. & Abbott, M. (2011). Smart-phone Augmented Reality for public
participation in urban planning. Proceedings of the 23rd Australian Computer-Human Interaction
Conference. ACM.
[10] Shen, Z. & Jiang, L. (2012). An augmented 3D iPad mobile application for communication,
collaboration, and learning (CCL) of building MEP systems. Computing in Civil Engineering, pp. 204-212.
[11] Yabuki, N., Hamada, Y. & Fukuda, T. (2012). Development of an accurate registration technique for
outdoor Augmented Reality using point cloud data. Proceedings of the 14th International Conference on
Computing in Civil and Building Engineering.
[12] Watzdorf, S. von & Michahelles, F. (2010). Accuracy of positioning data on smartphones.
Proceedings of the 3rd International Workshop on Location and the Web. ACM.
[13] Junaio (2013). Location Based Channels. Retrieved August 19, 2013 from
http://www.junaio.com/develop/docs/location-based-indooroutdoor/
[14] Wang, X., et al. (2012). A conceptual framework for integrating building information modeling with
Augmented Reality, Automation in Construction. Retrieved August 19, 2013 from
http://dx.doi.org/10.1016/j.autcon.2012.10.012
[15] Chi, H.-L., et al. (2013). Research trends and opportunities of Augmented Reality applications in
architecture, engineering, and construction, Automation in Construction. Retrieved August 19, 2013 from
Augmented Reality and Mixed Reality Technologies
STO-MSG-111 09 - 15
http://dx.doi.org/10.1016/j.autcon.2012.12.017
[16] CIOpGLO (2013). Virtual shooting simulators: fundamentals, applicability for military training and
a comparative study. Brazilian Army. Campinas, Brazil.
[17] Laser Shots (2013). Real combat: firearms training system. Product reference sheet. Orlando, United
States.
[18] Elbit Systems (2012). Land forces training systems: platforms and weapon operators training
systems. Product reference sheet. Haifa, Israel.
[19] Meggitt (2013). Small unit trainer simulation system. Product reference sheet. Suwanee, United
States.
[20] SAAB (2013). SAVIT: Small arms trainer. Product reference sheet. Huskvarna, Sweden.
[21] Nechvatal, J. (1999). Immersive ideals / critical distances: a study of the affinity between artistic
ideologies based in virtual reality and previous immersive idioms. 456 p. Thesis (PhD) - Centre for
Advanced Inquiry in the Interactive Arts. University of Wales College. Newport, Wales.
[22] Sensics (2013). High-performance augmented reality solution: technical overview and
specifications. . Retrieved August 19, 2013 from http://sensics.com/wp-
content/uploads/2013/05/Augmented-reality-solution-May-2013.pdf
[23] Project Glass (2013). Retrieved August 19, 2013 from https://plus.google.com/+projectglass/posts
[24] Inovega (2013). Retrieved August 19, 2013 from http://innovega-inc.com/new-architecture.php
[25] MetaVR (2013). Virtual Ft. Benning urban training site. Retrieved August 19, 2013 from
http://www.metavr.com/downloads/MetaVR_ FtBenning-Brochure.pdf
[26] Huilberts, Sander (2010). Captivating sound: the role of audio for immersion in computer games.
200 p. Thesis (PhD). Utrecht School of the Arts. Ultrecht, Netherlands.
[27] Dvice (2012). Augmented Reality glove lets you edit virtual objects in mid-air. Retrieved August 19,
2013 from http://www.dvice.com/archives/2012/06/augmented-reali-15.php
[28] SAAB Technologies (2008). Saab Training Systems: tactical engagement systemBT47. Product
reference sheet. Quadricon. Rio de Janeiro, Brazil.
[29] Kammerer, T. (2012). TargetHunter for AR.Drone. Retrieved August 19, 2013 from
https://itunes.apple.com/us/app/targethunter-for-ar-drone/id416855825?mt=8&ign-mpt=uo=4
[30] Simulogica (2012). SimIR: Interactive police and security training system. Product reference sheet.
Florianópolis, Brazil.
[31] VirTra (2013). Patented VirTra Threat-Fire return fire simulator. Retrieved August 19, 2013 from
http://www.virtra.com/threat-fire/
[32] Road to VR (2013). ARAIG: Gaming impact vest completes another piece of the vr puzzle, needs
your help on Kickstarter. Retrieved August 19, 2013 from http://www.roadtovr.com/2013/06/10/araig-
kickstarter-gaming-impact-vest-virtual-reality-6440
Augmented Reality and Mixed Reality Technologies
09 - 16 STO-MSG-111
[33] TN Games (2013). 3rd space: first person shooter gaming vest. Retrieved August 19, 2013 from
http://tngames.com/products
[34] Raytheon (2012). Virtual reality brings Hollywood star power to public safety. Retrieved August 19,
2013 from
http://www.raytheon.com/capabilities/rtnwcm/groups/ncs/documents/content/rtn_ncs_businesses_q22012_p
df.pdf
[35] Raytheon VIRTSIM demonstration (2012). Retrieved August 19, 2013 from
http://www.youtube.com/watch?v=aRWe9xsyReg
[36] Bimber, O. & Raskar, R. (2005). Spatial Augmented Reality: Merging Real and Virtual Worlds.
Wellesley: A K Peters, Ltd., 369.
[37] Milgram, P., et al. (1995). Augmented Reality: A Class of Displays on the Reality-Virtuality
Continuum. Proceedings of the SPIE Conference on Telemanipulator and Telepresence Technologies, 282-
292.
[38] Orrico, Alexandre. (2013) Realismo fantástico. Folha de São Paulo, Aug 19th 2013 (pp. F1). São
Paulo, Brazil.
[39] Macchiarella, N. D., Liu, D. & Vincenzi, D. (2009) A. Augmented Reality as a Mens of Job Task
Training in Aviation. In: Vincenzi, D. A. et al. Human factors in Simulation and Training. New York: 2009,
201-228. ISBN 9781420072839.
[40] Schmorrow, D. et al. (2009) Virtual Reality in the Training Environment. In: Vincenzi, D. A. et al.
Human factors in Simulation and Training. New York, 201-228. ISBN 9781420072839.
[41] Livingston, M. A., Rosenblum, L. J., Brown, D. G., Schmidt, G. S., Julier, S. J., Baillot, Y. &
Maassel, P. (2011). Military applications of augmented reality. In Handbook of Augmented Reality (pp. 671-
706). Springer New York. New York, USA.
[42] Oculus Rift (2013). Retrieved August 19, 2013 from http://www.oculusvr.com
[43] Kalkofen, Denis, Sandor, Christian, White, Sean & Schmalstieg, Dieter (2011). Visualization
techniques for augmented reality. In Handbook of Augmented Reality (pp. 65-98). Springer New York. New
York, USA.
[44] Kapp, K. M. (2012). The Gamification of Learning and Instruction: Game-based Methods and
Strategies for Training and Education. Pfeiffer. ISBN 1118096347.
[45] McIntyre, A. (2011). iPad and Beyond: What the Future of Computing Holds. Gartner. 30
September 2011. Retrieved August 19, 2013 from http://www.gartner.com/id=1812319
Augmented Reality and Mixed Reality Technologies
STO-MSG-111 09 - 17