Mobile Room Schedule Viewer using Augmented Reality
Transcript of Mobile Room Schedule Viewer using Augmented Reality
Code:________________
Faculty of Engineering and Sustainable Development
MOBILE ROOM SCHEDULE VIEWER USING
AUGMENTED REALITY
Yunyou Fan
June 2012
Bachelor Thesis, 15 credits, C
Computer Science
Study Programme for a Degree of Bachelor of Science in Computer Science
Examiner: Stefan Seipel
Supervisor: Peter Jenke
Mobile Room Schedule Viewer
Using
Augmented Reality
by
Yunyou Fan
Faculty of Engineering and Sustainable Development
University of Gävle
S-801 76 Gävle, Sweden
Email:
Abstract
This work aims to present a historical overview and analysis on the current technical details
within one category of Augmented Reality systems – Mobile Augmented Reality (MAR). In
addition, this research shows what practical and enjoyable mobile applications can be made
with MAR. A sample application has been developed for demonstration. The general aim of
this sample application is to develop a room number recognition system using Augmented
Reality technology. The work has demonstrated that an Android mobile phone equipped with
this sample application can overlay augmented room schedule onto the mobile screen. Several
experiments were carried out to evaluate the application. Recognition rate is an average of 91%
in continuous real time testing. The application is also tested for varying viewing angles as
well as different distances between the hand-held device and the targets to be tracked. Some
cases of failure have been identified and shown. Future work and results of an evaluation are
also discussed.
Contents
1 Introduction .............................................................................................................. 1 1.1 Definition ........................................................................................................................ 1 1.2 Objective ......................................................................................................................... 2
2 Related works ........................................................................................................... 3 2.1 Previous research ............................................................................................................. 3 2.2 Current AR based mobile applications ............................................................................ 5
2.2.1 Location-aware applications ................................................................................ 5 2.2.2 Context-aware applications .................................................................................. 6
3 Development of MAR demonstrator ...................................................................... 7 3.1 Overview ......................................................................................................................... 7 3.2 Implementation ................................................................................................................ 8 3.3 Experimental Results ..................................................................................................... 10
4 Discussion ............................................................................................................... 11 4.1 Hardware ....................................................................................................................... 13 4.2 Software ........................................................................................................................ 13 4.3 Privacy issues ................................................................................................................ 13
5 Conclusion .............................................................................................................. 14
6 Future work ............................................................................................................ 14
Acknowledgement ...................................................................................................... 15
Bibliography ............................................................................................................... 16
Appendix 1: Usability Survey for Further Evaluation ........................................... 19
Appendix 2: List of figures ........................................................................................ 21
Appendix 3: List of tables.......................................................................................... 22
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
1
1 Introduction
Augmented Reality (AR) technology has recently been a subject of much interest, and
has being used widely on PC such as 3D-game playing and information searching.
But now, thanks to the development of mobile hardware, this technology appears
destined for the mobile market, brought AR into everyday life.
1.1 Definition
There are several definitions in the literature for Augmented Reality. In 1994, Paul
Milgram and Fumio Kishino [1] provide taxonomy of mixed reality systems that
include Virtual Reality (VR) and Augmented Reality (AR) apart from Augmented
Virtuality (AV). Just a few years later, in 1997, Ronald Azuma [2] defines Augmented
Reality to be a technology that combines real environment surrounding with virtual
objects. Azuma notes that Augmented Reality shares three common characteristics:
Combines virtual characters with the actual world
Interactive in real time
Registered in 3D
The 2011 annual Horizon Report [3] published by the New Media Consortium
(NMC) describes Augmented Reality as the following:
“Augmented reality (AR) refers to the addition of a computer-assisted contextual
layer of information over the real world, creating a reality that is enhanced or
augmented.”
Figure 1. Milgram’s definition of virtuality continuum.
According to Milgram‟s definition of virtuality continuum [1](Figure 1), Mixed
Reality (MR) environments are those in which real world and virtual world objects are
presented together on a single display. Virtual Reality (VR) simulates physical
presence in places in the real world with computer-simulated environments.
Augmented Virtuality is a term that merges of actual world objects into virtual worlds.
Unlike Virtual Reality creating a simulation of reality and Augmented Virtuality using
virtual worlds replace reality, Augmented Reality enhance the user experience by
projecting the virtual objects onto the observer‟s perception. In other words, with
Augmented Reality, virtual objects only supplement the real world, the real world
objects are still in the real world.
Combined with handheld displays, cameras, acceleration, compass, GPS sensors,
Mobile Augmented Reality has been one of the fastest growing research areas in
Augmented Reality, partially due to blending virtual imaging into the video stream of
a mobile device's camera in real-time and ubiquitous platforms for supporting MAR.
The MAR technology can be applied widely, such as in navigation, situational
awareness, and geography located information. Applications in practice include a
Mixed Reality
Real Environment
Augmented Reality
Augmented Virtuality
Virtual Environment
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
2
university information system [4] [5], shooting game [6] and car finder with GPS [7],
and so on.
Based on how augmented information is aligned with actual world, there are two
primary types of MAR implementations: Markerless and Marker-Based.
The key difference in Markerless AR is the method used to place virtual objects
in the user‟s view. Markerless AR typically uses the GPS feature of a smartphone to
locate and interact with AR resources. Markerless AR is often more reliant on the
capabilities of the device being used such as the GPS location, velocity meter, etc.
With Markerless AR, no special markers are required. The software is able to
recognize natural features, and more complex visual information. Aside from
expanding the scope of what can be recognized and in turn augmented, it also opens
up increased opportunities to create more natural interactions between digital and
physical world elements which have the potential for a wide variety of applications.
However, tracking and registration techniques are much more complex in Markerless
AR. Drawbacks of Markerless AR including real/virtual image discrepancy, system
calibration, or registration problems such as observers fail to adjust their perceptions
have to be overcome.
When location data isn‟t used, a marker is often used. Traditional markers, for
example, data matrix codes and QR codes, need special black and white regions or
codes to be recognized while Image Targets do not. However, in order to have more
features to be found, high local contrast is needed. A Marker-based implementation
utilizes the traditional marker (see Figure 3), such as QR Code/2D barcode to produce
a result when it is sensed by a reader, typically a camera mounted on the mobile
screen. Unlike traditional markers, Image Targets do not need special black and white
regions to be recognized. The Vuforia SDK [8] uses sophisticated algorithms to detect
and track the natural features that are analyzed in the target image itself. It recognizes
the Image Target by comparing these natural features against a known target resource
database. Once the Image Target is detected, the SDK will store these extracted
features in a database and then compared at run-time with features in the live camera
image. Targets that are accurately detected should be rich in detail, have good contrast,
must be generally well lit and not dull in brightness or color and should not have
repetitive patterns. Figure 2 is an example of a bad image target that has repeated
pattern that cannot be distinguished by the detector, small yellow crosses that appear
around the tree shows the features can be used to match images.
Figure 2. An example of a bad image target.
Marker-based AR is generally easier to build thus it becomes more prevalent in
the mobile application market.
1.2 Objective
The objective of this research is to provide an overview of the previous research as
well as the current applications in the field of Mobile Augmented Reality. A sample
application for the Android platform is developed using AR technology for
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
3
demonstration and evaluation. The general aim of this sample application is to develop
a room recognition system to act as a room schedule viewer for University of Gavle.
The digits recognition is achieved by OCR (optical character recognition). In
particular, this sample application is context-aware and is able to tell both teachers and
students whether the room is free or occupied by loading schedules for the specific
room published in KronoX. This also allows a comparison between an AR based
application and a non-AR based application.
2 Related works
2.1 Previous research
In 1995, Jun Rekimoto and Katashi Nagao started to experiment with handheld AR
based on the color codes detection method [9]. One year later, Rekimoto [10]
proposed his new method using a 2D matrix barcode (see Figure 3) as landmarks for
both object identification and registration when producing an Augmented Reality
system. It was one of the first marker systems to enable the use of a 2D matrix barcode
to identify a large number of objects.
Figure 3. 2D matrix marker.
The first outdoor Mobile Augmented Reality System (MARS), Touring Machine
(see Figure 4) [11] was presented by Feiner et al. in 1997 featured tracking in a
handheld display with touchpad interface. The system observed the environment
through an HMD (Head Mounted Display), and showed additional information such as
the names of buildings, historical events at the observed locations.
Figure 4. Touring Machine [12].
Five years later, authors demonstrated an AR restaurant guide on their Touring
Machine. Information such as reviews, menus, and website for nearby restaurants
were overlaid onto the display [13] [14] .
ARToolKit [15] is a free and open source library under GPL license for
developing AR applications. Similar to Rekimoto‟s method, it is a flexible pose
tracking library allowing six degrees of freedom (6DoF) movement in 3D space. That
is, controllers are free to move in 3-dimensional direction. The system uses a template
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
4
matching approach and square markers for object recognition, and achieves AR using
a video or webcam. The project was launched in 1999 and is still very popular among
AR developers. A number of well-known AR libraries, including ARTag [16], AndAR
[17], NyARToolkit [18] and FLARToolKit [19], are inspired by ARToolkit. Wagner
and Barakonyi [20] created the first self-contained AR marker tracking system on a
PDA using ARToolKit.
In the same year, Höllerer et al. [21] presented their MAR system that allowed
visitors to experience Augmented Reality outdoors and receive a guided campus tour
with additional 3D overlay on top of the real world. This was the first MAR system to
use RTK GPS and an inertial-magnetic orientation tracker.
In 2000, Bruce Thomas et al. presented ARQuake [22], an extension to the
popular desktop game Quake, and showed how it could be used for outdoor gameplay.
ARQuake was a first-person perspective application based on a 6DOF tracking system
[23]. Users equipped a HMD on the head, and a wearable computer, ran around in the
real world whilst playing a game in the virtual world. The game could be played in- or
outdoors where the usual keyboard and mouse commands for movements and actions
were performed by movements of the user in the real environment and using the
simple input interface. ARQuake was the first fully working Augmented Reality game
created for outdoor use.
CyberCode [24], introduced in 2000, was a conventional tagging system which
allowed the computer to track position and orientation in three dimensions space.
System was based on the 2D barcode technology and could be used to determine the
3D position of the tagged objects.
Newman et al. presented the Bat system [25], a PDA-based MAR system in the
year 2001. Bat system used two different ways to build an indoor AR system. The first
method used an HMD connected to a laptop. The second method used a PDA with a
fixed configuration to provide a convenient portal with which the user can quickly
viewed the augmented information on the PDA‟s screen.
Reitmayr et al. [26] introduced a mobile collaborative augmented reality system
in 2001. The ideas of mobile computing and collaborative augmented reality were
combined and merged into a single augmented system. The communication between
users was done through LAN and wireless LAN (WLAN).
In 2008, Mobilizy launched Wikitude World Browser [4], an application that
combined position data with Wikipedia entries. In 2011, Wikitude announced the
Wikitude ARchitect Engine at the ARE conference in Silicon Valley [27]. ARchitect
is very flexible; developers are allowed to engage directly with their users on their
terms. A similar step up, Layar [5], works by retrieving digital information over a
network connection and augmented onto top of the screen with a combination of
devices‟ built-in sensors.
Sony is doing a project called SmartAR [28] which was published in 2011. It is a
great experimental demonstration of how the technology is evolving with Markerless
AR. According to Sony, SmartAR (see Figure 5) makes use of the Markerless
approach by combining the traditional object recognition technology with their
proprietary probabilistic matching method in the 3D space. With this method, the 3D
overlay will be retrieved from the database with minimal calculations when
recognizing an object from actual world.
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
5
Figure 5. Sony develops “SmartAR” integrated AR technology.
2.2 Current AR based mobile applications
Mobile phones have becoming a part of almost everyone‟s lives recently. Indeed, AR
based mobile applications have made way to a lot of comfort and convenience to a lot
of mobile phone users already, partially due to people always want to know all the
details about their interests. Such applications cover a wide variety of technologies,
devices and goals.
I‟ve grouped the AR based mobile applications into two categories, location-
aware applications and context-aware applications, based on its key features.
2.2.1 Location-aware applications
Nowadays, location-aware technologies, like assisted GPS satellite service, video
camera, and digital compass allow people to share their real-time location with friends
and family on their mobile devices [29] [30].
Current MAR systems typically use the devices‟ built-in sensors, like camera,
compass and accelerometer to determine the direction of view, working with GPS
tracking service or Network Location Provider, to assign the augmented information
to the obtained location coordinates. The location information can be obtained from
GPS service or Network Location Provider. However, GPS is limited to outdoor use
only while the Network Location Provider would be suitable for both indoor and
outdoor use. Here, camera captures the world as seen through its lens and shows it on
the screen; GPS and Network Location Provider obtain the observer‟s location and
compass and accelerometer determine the field of view.
2.2.1.1 How it works
It is a simple process that involves four basic tasks:
#Step one: Obtaining user location
Figure 6. Obtaining user location.
Return The Location
Acquire User
Location
GPS/
Location
Provider
Mobile
devices
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
6
#Step two: Request/receive observed location information from ISP (Information
Service Provider)
Figure 7. Request/receive observed location information from ISP.
#Step three: Determine the lens direction using built-in sensors (e.g. gravity
sensor, digital compass).
#Step four: The camera captures the world as seen through its lens and shows the
receiving updates about users‟ surroundings on the screen.
Figure 8. Combiner.
While MAR is emerging, it is currently rather limited. Most of these Location-
based applications rely heavily on GPS Services. But GPS itself is limited due to
many factors - accuracy, usually between ten and thirty meters, only works outdoor.
Besides the GPS limitation, mobile network requirements, the limited battery power,
as well as camera preview callback memory issues can affect the usefulness of these
location based application.
Another important issue to think about is the ability to continue operation in case
the communication with the tracking system is temporarily lost.
Location-aware applications might touch upon privacy issues, since it allows
people to check where a person is without consent. The problems (e.g. privacy issues)
are discussed in a later section.
2.2.2 Context-aware applications
Context-aware application can be considered as a complement to location-aware
application since context can be applied more flexibly with mobile computing with
any moving entities. For instance, wireless subscribers can provide real-time
information about observers‟ surroundings, overlay information about locations,
surrounding buildings and friends who might be nearby, and these information is
stored and shared with others via online services (see Figure 9).
These activities result in detailed maps and allow additional information that
otherwise cannot be identified by visual perception to be displayed on the mobile
devices. Objects captured by camera can be quickly recognized and tracked at high-
speed along with the movement of the camera, as it is displayed over the actual 3D
space [28].
Receive Data from
ISP
Send a Request to
ISP
Mobile Device
Information Provider
Location Provider
Mobile
devices
ISP
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
7
Figure 9. Looking at the world through AR [31].
When location data isn‟t used, a marker is often required. Marker-based AR
applications are the most common in the AR market. According to Azuma‟s definition
of AR, the ability of interacting in real time is important since it makes the AR not as
simple as an information rendering tool. But it is in terms of increasing reality in real
time where the prospects seem the most impressive. AR based mobile applications
built with Vuforia SDK [8] must have a known target dataset and good features that
can be used to match images in the camera view. In some cases registration problems
might happen since real scene coordinate system and virtual object coordinate system
should be aligned with each other. Registration problems can cause observers fail to
adjust their perceptions.
Perhaps this is the future to wear glasses. The idea of eyewear has been put into
practice by Vuzixs who offers glasses Wrap 920AR [32] to display 3D images in the
real world. Similar to Vuzixs, Nokia was showing off its own alternative to use
camera supported mobile phones as the platform for sensor-based, video see-through
mobile augmented reality [33]. At the beginning of the year 2012, Google has
announced Project AR Glass [34] [35].
3 Development of MAR demonstrator
Imagine a mobile phone is a lens into the real world that shows the schedule at the
University of Gävle by displaying virtual information on the top of reality – users are
allowed seeing that with the real world environment right in front of them. This is the
key of Mobile Augmented Reality and also is what I am trying to illustrate.
3.1 Overview
The project is developed using Eclipse with a plugin called ADT (Android
Development Tools). Eclipse is appreciated since it can directly invoke the tools that
needed while developing applications. The Tesseract OCR Engine [36], the ZXing [37]
library for image processing, Android SDK, NDK (a companion tool to the Android
SDK that allows building application in native code), Cygwin and Git are also used to
compile the project. The application of the framework is implemented in Java
language (Oracle‟s JDK7u4).
Environmental information will be obtained via mobile camera. The video of the
real world surroundings and the images are combined, and then displayed on the
screen. Figure 10 shows the concept of how the system is built.
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
8
Figure 10. The conceptual diagram of Mobile based AR.
3.2 Implementation
A procedure to carry out the task is as follows in Figure 11. The system comprises of
three subparts, Scene generator, Virtual objects generator and the Combiner.
Figure 11. System architecture.
The details of the procedures are described below:
#Step one: Real world generator
The user is required to predefine the position of the number plate using a
bounding box and then the application will automatically inspect the room number
within the given area in order to recognize the room number. Camera data is provided
by the android framework through the Camera API and Camera Intent. The common
method to custom a surface view where the live camera preview will be loaded is
using the class SurfaceView.
In order to be more feasible and effective, I set up a target using model-based
recognition. In the campus of University of Gävle, room numbers, composed by five
digits, together with “:”, with a clear plastic cover are printed on the wall. If only one
letter (A-Ö) is detected, it is rejected since it is not part of target string and thus cannot
be a room number. A whitelist “0123456789:” is created for rejection. The use of
whitelist can simplify the digit recognition and improve quality of recognition while
achieving good performance. The room number model is shown in Figure 12.
User
Combiner
Android device video camera on
mobile device
Scene
generator
Tracker
Room Number
Recognition
Scheme
information
video of real world
surroundings
Real world
generator
• Prepping the camera
• OCR by running the Tesseract OCR engine to recognize room number in images captured by the device's camera.
Virtual objects
generator • Using web services(KronoX)
Combiner • Achieve a real-time display
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
9
Figure 12. Room number model.
As the images differ from each other, it is not possible to use a static global
adaptive threshold because there could be a certain amount of loss of information.
Thus, image pre-processing is required as a prerequisite to the OCR. The Tesseract
OCR Engine consists of a main program, a word level recognizer, a low-level
character classifier and one or more dictionaries. The word level recognizer is a
module that organizes text into lines and words. The character classifier works with
dictionaries to classify the words. The digit recognition is achieved by a numbers of
outline comparisons between characters for separation, and then applying a Prototype-
Feature-Matching process on these extracted outline fragments. Each scaled character
will be matched once with each template, and the one with the highest confidence will
be the recognition result. However, even the highest confidence can still very low
sometime. The last step is a combination of broken characters. In order to identify the
room number on the Android platform, Android APIs for the Tesseract library are
used [38] [39], which provides a set of Android APIs for accessing natively-compiled
Tesseract OCR libraries. However, some failure of character recognition may happen;
this will be discussed in the later section.
#Step two: Virtual objects generator
Since the augmented information is the schedule website, the processing of
virtual objects generator can be regarded as the processing of link generator which is a
made up of fixed URL and recognized the room number ( Figure 13 ).
Figure 13. Link generator.
#Step three: Combiner
The final step is to bring the camera view and augmented schedule together onto
the observer‟s perception. This step is a process of registration which is vital for the
AR. AR requires accurate registration of virtual objects in order to render them into
the actual world. As mentioned before, the magic of AR is the idea that drawing
something on top of it. For mobile devices, this process is as simple as draw
something over the camera and can be accomplished by various ways. In this
application, the magic is done by adding two views to the selected layer. In order to
achieve a real-time display, the WebView class is used to ensure recognition result
and web page is shown at the same time.
Schedule Link
Fixed URL Room number
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
10
The running result is shown in Figure 14. The TextView (blue) displays the
room number that interested, the WebView showing the activities for the room.
Figure 14. Running result.
3.3 Experimental Results
I have conducted experiments in order to examine the performance of the application
under varying conditions and to validate the feasibility of AR based room recognition
system in real time. The experiments were carried out under a natural light condition
in the building 99, University of Gävle. These experimental results are only in case of
recognizing specific room number „99:518‟.
The system was tested in real time and gave reliable room recognition. The major
achievement of this work is the recognition rate of correct identification, which is 91%
in continuous real time testing. The recognition rate of correct identification varied
while capturing images in the different angle of view and in the different distance
between observer and door.
The application is tested on over the different angle of view as well as the
different distance between observer and door. While capturing these images, the angle
of view and the distance between observer and door varied according to the
experimental setup. These images were subjected to pre-processing which comprises
of some standard image processing algorithms. The resultant images were then fed to
the proposed OCR system.
The first experiment measures the correct identification while capturing images
in the different angle of view. The result is shown in Table 1 and Figure 15. In Table 1,
recognition rate refers the number of the successful rate in which the application can
recognize room number correctly. The failure is considered to have been caused by
errors in the camera orientation. So the angle of view is crucial for room number
recognition.
Table 1. Recognition rate according to different angle of view.
(deg)
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
11
Figure 15. Recognition rate according to different angle of view.
The second experiment measures the correct identification while capturing
images in the different distance between observer and door. The result is shown in
Table 2 and Figure 16. In Table 2, recognition rate is the number of the successful rate
in which the room numbers were recognized correctly. As the results suggested,
normally when the distance between observer and door was bigger than 70 cm, the
digits were too small to be recognized. The recommend distance of the observer to the
door should be between 10 and 30 cm, not more than 50 cm.
Table 2. Recognition rate according to different distance.
93 83 41
Figure 16. Recognition rate according to different distance.
4 Discussion
From the experiment results, we can see that the application gives no reliable
recognition result if room number is distorted by perspective projection when the
angle of view θ is small or the distance of the observer to the door is large. False
recognition caused by errors in the angle of view is most likely due to a lack of
resolution in depth. Thus lower accuracy is often unavoidable. Any image taken at a
slanted angle will thus affect the recognition. With an increase of angle of view, the
0
20
40
60
80
100
40 50 60 90
recognition rate (%)
θ(deg)
θ(deg)
θ(deg)
0
20
40
60
80
100
10 30 50 70
recognition rate (%)
distance(cm)
distance(cm)
distance(cm)
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
12
image will become distorted. This problem is hard to handle because it very much
depends on the camera's depth of field and focus settings. The problem is even more
serious here since the plastic covering on the room number plate can cause reflections.
In Figure 15, there is a sudden drop between 50 to 60 degrees, which is mainly caused
by reflections. In this area, the amount of light reflected increases as the angle of view
increases. While the room numbers are at a quite distance from the user, digits are too
small and can be easily wrongly recognized. To overcome this, an alternate approach
is to use the camera phone featuring a high zoom ratio.
Besides, the recognition of room number is also affected by factors such as the
ambiguity of an image, motion blur effect and brightness level. Ambiguity makes an
image can easily be wrongly recognized (Figure 17). Motion blur (Figure 18) is
partially due to hand shaking. Blurred or ghostly images make it difficult to perform
recognition on. Handle motion blur from both hand shaking and object moving can be
minimized through several techniques such as separating individual frames from a
video stream.
Figure 17. The ambiguous problem.
Figure 18. The motion blur problem.
Brightness level is another crucial problem in room number recognition. Many
other problems are more or less subdued by its lack of suitable light. The experiments
are carried out under a natural light condition, but perhaps lighting conditions in the
surrounding still not enough. Low lighting conditions causes the input image to be
unclear and consists mainly of dark pixels which seriously affect the recognition. In
our case, local contrast should not one of the major problems that limiting the
recognition rate since the sample application only handle the black room numbers
printed on the white wall. However, when the angle of view θ is small or observer and
door is quite a distance away, room numbers are not be distinct enough for accurate
recognition. An input image has high local contrast features and yield easier to be
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
13
recognized. The higher contrast the input image has, the better OCR engine works.
Without a good local contrast, the shape and outline of each digit will be ambiguous.
Given all above problems, it is impossible to achieve 100% recognition rate. In
order to minimize the effects of above problems, an appropriate image pre-processing
could be applied on the original image before being processed by the OCR engine.
The most common techniques used to improve the brightness and contrast of an image
is histogram equalization where one attempts to map one intensity distribution of the
original image histogram to another wider and more uniform distribution so that the
intensity values can spread over the whole intensity range. This is done by distributing
the intensities of the input image from dark to bright, for example, the number of
pixels for each luminous intensity. After these improvements are applied, the problem
when the distributions of the original histogram are very close to each other could be
solved. All in all, lighting conditions should be carefully controlled to avoid
reflections and shadows.
As described in chapter 3.2, the Tesseract OCR engine verifies the recognized
number by voting the recognition result, the one with the highest confidence is
considered as a correct room number. In the experiment, the highest confidence is still
very low due to errors in the angle of view as well as the distance of the observer to
the door, which leads an increasing of false recognition rate. So it is risky to use the
highest confidence result without any filtering. Rejection is an alternative to evaluate
the reliability of the resultant data. This can be done by setting a threshold value for
the acceptable highest confidence result - if the highest confidence is lower than
certain threshold value, then it is rejected and will not assume to be a correct room
number. In this way, the false recognition rate could be reduced.
4.1 Hardware
The sample application is compatible with Android 2.2 and up. It doesn‟t working on
the older Android OS such as Android 2.1 and Android 1.6. This is because the
application uses YUV image format API to speed up compression and decoding.
The YUV data only supports Android 2.2 and up.
Most Augmented reality applications require the image processing which is
generally very computationally expensive. What‟s more, there is a limitation of
memory allocation for each process max memory use, it is normally 24 MB, but on
some older devices, the limit is even lower at 16 MB. The memory used by Bitmaps is
also included in the limit. When developing an AR application, it is pretty easy to
reach this limit and get the OutOfMemoryError exception. The problem is finally
solved by allocating memory from native code using the NDK. However, as new
Android OS get released, users will probably see more AR applications with cool
features.
4.2 Software
In the sample application, text drawings might suffice which means rendering is not
currently one of the critical problems. However, Mobile AR systems must be worn,
which challenges developers to minimize weight and maintain performance. One
disadvantage of augmented reality applications is the information overloading. This
occurs when too much information is offered than the user is able to process.
Reducing information overloading can be complicated; one approach is to reduce the
rate at which new updates appear by specifying an interval time.
4.3 Privacy issues
The sample application is context-aware and does not require any extra permission
(besides internet access permission and the use of camera permission). But usually,
most AR-based applications have access to much permissions, such as allow hardware
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
14
controls(take pictures and videos), full internet access, coarse (network-based)
location, fine (GPS) location, modify and delete USB/SD card storage contents, and
view and change network/Wi-Fi state [4] .
Possible risks include:
enabling the GPS and then streaming those feeds to a remote server
sending email/SMS/MMS(services that cost you money) without user‟s
confirmation
read personal information and send these data to other people
linking to unsafe websites
These application and services may run in the background and users didn't even
realize they are still running.
5 Conclusion
In this paper, I have presented an analysis on the current technical details and the
practical use in the field of Mobile Augmented Reality. Mobile Augmented Reality
technology offers very unique experiences for users to interact with surroundings, and
the use of mobile phones with MAR allows these experiences to be extended with
visibility. Moreover, I have developed a sample application that can simultaneously
recognize room number by means of optical character recognition using Mobile
Augmented Reality technology. The MAR demonstrator achieves the task that
superimposes augmented schedule information onto the mobile screen. Several
experiments were carried out to evaluate the application. A few problems in room
number recognition have been identified, including varying viewing angles and
different distances between observer and door. The experimental results showed room
numbers can be correctly recognized by using the MAR demonstrator. The recognition
rate was an average of 91% in continuous real time testing. This has been proved
through several experiments.
Despite their limited computation power, I think mobile device is the perfect
platform for who want a truly immersive augmented reality experience. Most of
mobile devices are equipped with a digital camera on the back, which provides a
nature way to achieve see-through effect at affordable cost.
6 Future work
According to the three common characteristics of Augmented Reality offered by
Azuma, the interaction between users and virtual information play an essential role in
augmented reality. Therefore it is important to evaluate user experience when using an
AR-based application and come up with further improvement.
Later, an experiment could be carried out for evaluation. The purpose of this
experiment should focus on the different user reactions when using an AR based
application and a non-AR based application (Figure 19).
I have offered a Usability Survey (Figure 20) (see Appendix 1) which is a part of
the application. The possibility of future evaluation was considered at the application
design stage.
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
15
Figure 19. GUI and Non-AR function.
Figure 20. Usability Survey for future evaluation.
Acknowledgement
I would like to thank Peter Jenke for his help of my research project.
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
16
Bibliography
[1] Paul Milgram and Fumio Kishino, "Taxonomy of Mixed Reality Visual Displays",
IEICE Transactions on Information Systems, pp. 1321-1329, December 1994.
[2] Ronald T. Azuma, "A Survey of Augmented Reality", Presence: Teleoperators and
Virtual Environments, pp. 355-385, August 1997.
[3] The New Media Consortium and the EDUCAUSE Learning initiative, "Two to
Three Years: Augmented Reality", The 2011 NMC Horizon Report, pp. 16-20,
February 2011.
[4] Wikitude GmbH. (2012, March) Wikitude World Browser. [Online].
https://play.google.com/store/apps/details?id=com.wikitude&hl=en
[5] Layar. Layar. [Online]. http://www.layar.com/
[6] ProjectX. (2012, March) AR Shooting. [Online].
https://play.google.com/store/apps/details?id=com.project_x.ARShooting&hl=sv
[7] Hexter Labs Software. (2012, January ) Car Finder AR. [Online].
https://play.google.com/store/apps/details?id=net.androoid.carxare&feature=search_
result#?t=W251bGwsMSwxLDEsIm5ldC5hbmRyb29pZC5jYXJ4YXJlIl0.
[8] Qualcomm. Augmented Reality (Vuforia) - Qualcomm Developer Network.
[Online]. https://developer.qualcomm.com/develop/mobile-technologies/augmented-
reality
[9] Jun Rekimoto and Katashi Nagao, "The world through the computer: computer
augmented interaction with real world environments", in 8th annual ACM
symposium on User interface and software technology(UIST '95), 1995, pp. 29-36.
[10] Jun Rekimoto, "Matrix: A Realitime Object Identification and Registration Method
for Augmented Reality", in 3rd Asia Pacific Conference on Computer Human
Interaction(APCHI 98), Kanagawa, 1998, pp. 63 - 68.
[11] S. Feiner, B. MacIntyre, T. Höllerer and A. Webster, "A Touring Machine:
Prototyping 3D Mobile Augmented Reality Systems for Exploring the Urban
Environment", in 1st International Symposium on Wearable Computers(ISWC '97),
Cambridge, MA, 1997, pp. 74-81.
[12] Steven Feiner, Tobias Höllerer, Elias Gagas, Drexel Hallaway, Tachio Terauchi,
Sinem Güven, and Blair MacIntyre. MARS - Mobile Augmented Reality Systems.
[Online]. http://graphics.cs.columbia.edu/projects/mars/mars.html
[13] B. Bell, S. Feiner, and T. Höllerer, "View Management for Virtual and Augmented
Reality", in UIST '01, Orlando, 2001 , pp. 101-110.
[14] B. Bell, T. Höllerer, and S. Feiner, "An Annotated Situation-Awareness Aid for
Augmented Reality", in UIST '02, Paris, 2002, pp. 213-216.
[15] Hirokazu Kato and Mark Billinghurst, "Marker tracking and HMD calibration for a
video-based augmented reality conferencing system", in 2nd International Workshop
on Augmented Reality (IWAR'99), San Francisco, 1999, pp. 85-94.
[16] Mark Fiala. (1999, February ) ARTag. [Online]. http://www.artag.net/
[17] ARToolWorks. andar - AndAR - Android Augmented Reality - Google Project
Hosting. [Online]. http://code.google.com/p/andar/
[18] ARToolWorks. (2008, May) NyARToolkit project. [Online].
http://nyatla.jp/nyartoolkit/wp/
[19] Saqoosha amd nyatla. (2008, october) saqoosha/FLARToolKit/en. [Online].
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
17
http://www.artoolworks.com/products/web/flartoolkit-2/
[20] Daniel Wagner and Istvan Barakonyi, "Augmented Reality Kanji Learning", in 2nd
International Symposium on Mixed and Augmented Reality(ISMAR 2003), Tokyo,
2003, pp. 335-336.
[21] T. Höllerer, S. Feiner, and J. Pavlik, "Situated documentaries: Embedding
multimedia presentations in the real world", in 3rd International Symposium on
Wearable Computers(ISWC'99), New York, 1999, pp. 79-86.
[22] Thomas, B., Close, B., Donoghue, J., Squires, J., De Bondi, P., Morris, M., and
Piekarski, W, "ARQuake: An Outdoor/Indoor Augmented Reality First Person
Application", in 4th International Symposium on Wearable Computers(ISWC'10),
October 2010, pp. 139-146.
[23] J. Park, B.Jiang, and U.Neumann, "Vision-based pose computation: Robust and
accurate augmented reality tracking", in 2nd IEEE and ACM International Workshop
on Augmented Reality (IWAR '99), San Francisco, California, 1999, pp. 3-12.
[24] Jun Rekimoto and Yuji Ayatsuka, "CyberCode: designing augmented reality
environments with visual tags", in ACM Conference on Designing Augmented
Reality Environments (DARE 2000), New York, 2000, pp. 1-10.
[25] J. Newman, D. Ingram and A. Hopper, "Augmented Reality in a Wide Area Sentient
Environment", in 2nd IEEE and ACM International Symposium on Augmented
Reality (ISAR 2001), New York, 2001, pp. 77-86.
[26] G. Reitmayr, and D. Schmalstieg, "Mobile Collaborative Augmented Reality", in
2nd International Symposium on Augmented Reality (ISAR 2001), New York, 2001,
pp. 114-123.
[27] Wikitude. (2011, September ) Wikitude reveals the ARchitect Developer Kit.
[Online]. http://www.wikitude.com/enwikitude-reveals-architect-developer-kit
[28] Sony Corporation. (2011, May) Sony develops “SmartAR” Integrated Augmented
Reality technology. [Online]. http://www.sony.net/SonyInfo/News/Press/201105/11-
058E/index.html
[29] G. Takacs, V. Chandrasekhar, S. Tsai, D. Chen, R. Grzeszczuk, and B.Girod,
"Unified real-time tracking and recognition with rotation-invariant fast features", in
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2010,
pp. 934-941.
[30] D. Chen, S. Tsai, R. Vedantham, R. Grzeszczuk, and B. Girod, "Streaming mobile
augmented reality on mobile phones", in 8th International Symposium on Mixed and
Augmented Reality (ISMAR 2009), Orlando, October 2009, pp. 181-182.
[31] The Big Idea: Augmented Reality. [Online]. http://ngm.nationalgeographic.com/big-
idea/14/augmented-reality
[32] Vuzix Corporation. (2010, October) Wrap 920AR. [Online].
http://www.vuzix.com/consumer/products_wrap920ar.html
[33] Nokia. (2009) Nokia Mixed Reality. [Online]. http://research.nokia.com/page/695
[34] Bilton Nick. (2012, April) Google Begins Testing Its Augmented-Reality Glasses -
NYTimes.com. [Online]. http://bits.blogs.nytimes.com/2012/04/04/google-begins-
testing-its-augmented-reality-glasses/
[35] NICK BILTON. (2012, February) Google to Sell Heads-Up Display Glasses by
Year‟s End. [Online]. http://bits.blogs.nytimes.com/2012/02/21/google-to-sell-
terminator-style-glasses-by-years-end/
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
18
[36] Google. (1995) The Tesseract OCR engine. [Online].
http://code.google.com/p/tesseract-ocr/
[37] ZXing. (2007, November) zxing - Multi-format 1D-2D barcode image processing
library with clients for Android, Java. [Online]. http://code.google.com/p/zxing/
[38] Alan Viverette. Android APIs for the Tesseract OCR library. [Online].
http://code.google.com/p/tesseract-android-tools/
[39] Robert Theis. (2011) Tess-two. [Online]. https://github.com/rmtheis/tess-two
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
19
Appendix 1: Usability Survey for Further Evaluation
Has the application run successfully? * 1.
a. Yes
b. No
Do you find the information you are looking for (room scheduling)? * 2.
a. Yes
b. No
Please state any difficulties you had when using the application, if any. 3.
a. can't connect to the internet
b. can't recognize the room number correctly
c. don't know how to use this application
d. other
Compared with non-AR based application I have used, I found an AR based 4.
application is* (concerning the OPERABILITY)
a. easier to use
b. about average
c. more difficult to use
Compared with non-AR based application I have used, I found an AR based 5.
application is* (concerning the EFFECTIVENESS)
a. take more time to find the information
b. take the same time to find the information
c. take less time to find the information
Compared with non-AR based application I have used, I found an AR based 6.
application is* (concerning the USEFULNESS)
a. more helpful
b. about the same
c. less helpful
Have you ever used any AR based application on your mobile phone? * 7.
a. Yes
b. No
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
20
c. Never Heard of
Any comment? 8.
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
21
Appendix 2: List of figures
Figure 1. Milgram‟s definition of virtuality continuum. ---------------------------------- 1
Figure 2. An example of a bad image target. ----------------------------------------------- 2
Figure 3. 2D matrix marker. ------------------------------------------------------------------- 3
Figure 4. Touring Machine [12]. ------------------------------------------------------------- 3
Figure 5. Sony develops “SmartAR” integrated AR technology. ------------------------ 5
Figure 6. Obtaining user location. ------------------------------------------------------------ 5
Figure 7. Request/receive observed location information from ISP. -------------------- 6
Figure 8. Combiner. ---------------------------------------------------------------------------- 6
Figure 9. Looking at the world through AR [31]. ------------------------------------------ 7
Figure 10. The conceptual diagram of Mobile based AR. -------------------------------- 8
Figure 11. System architecture. --------------------------------------------------------------- 8
Figure 12. Room number model. ------------------------------------------------------------- 9
Figure 13. Link generator. --------------------------------------------------------------------- 9
Figure 14. Running result. -------------------------------------------------------------------- 10
Figure 15. Recognition rate according to different angle of view.---------------------- 11
Figure 16. Recognition rate according to different distance. ---------------------------- 11
Figure 17. The ambiguous problem. -------------------------------------------------------- 12
Figure 18. The motion blur problem. ------------------------------------------------------- 12
Figure 19. GUI and Non-AR function. ----------------------------------------------------- 15
Figure 20. Usability Survey for future evaluation. ---------------------------------------- 15
Yunyou Fan Mobile Room Schedule Viewer Using Augmented Reality
22
Appendix 3: List of tables
Table 1. Recognition rate according to different angle of view. ................................ 10
Table 2. Recognition rate according to different distance. ........................................ 11