TEO TZONG REN - UNIMAS Institutional Repository phone augmented reality business... · MOBILE PHONE...
Transcript of TEO TZONG REN - UNIMAS Institutional Repository phone augmented reality business... · MOBILE PHONE...
MOBILE PHONE AUGMENTED REALITY BUSINESS CARD
TEO TZONG REN
'I'bis prroject is submitted in partial fülfilmcnt of the requirements für a Kachelur oof. Science with honours
(('ogniti\e Sciences)
Faculty of ('ognitivc Sciences and I luman Development l! NIVI: RSI"fl MALAYSIA SARAWAK
(2010)
ACKNOWLEGEMENT
First and fbremost I offer my sincerest gratitude to my supervisor, Dr. Edmund Ng Giap Weng, who has supported me throughout my project with his patience and knowledge whilst allowing me the room to work in my own way. It would have been next to impossible to complete this project without his help and guidance. I would also like to thank Md. Abdullah Al Jubair, who assisted me in developing the project's application. Next, I would like to thank my parents fir
supporting me throughout all my studies at University. Lastly, I offer my regards and blessings to all of those who had supported me in any respect throughout this
project's completion.
III
TABLE OF CONTENTS
Acknowledgement Table of Contents List of Figures List of Tables Abstract Ahstruk
Chapter I- INTRODU('TION
1.0 Overview 1.1 Background 1.2 Problem Statement 1.3 Objective
1.3.1 General Objective 1.3.2 Specific Objectives
1.4 Importance of'Research 1.5 Project Scope 1.6 Values ot'Study 1.7 Significance of'Study 1.8 Structure of Project 1.9 Conclusion
Chapter 2- LI'T'ERATURE REVIEW
Page
iii iv vii x xi xii
i i 3 4 4 4 4 5 5 6 6 7
2.0 Overview K 2.1 Introduction of'Augmented Reality K 2.2 Introduction of'Mobile Phone Augmented Reality IO
2.2.1 S mhian S60 II 2.2
.2 S60Augmented Reality The Map Tracking II
2.2.3 History of'Mobile Augmented Reality 13 2.3 Introduction of'Business Card 15
2.3.1 Most Typical of Business Card Formats 16 2.3.2 Current Research on Augmented Business Card 17
2.4 Mobile 3D Map 18 2.5 Conclusion 19
IV
Chapter 3- METHODOLOGY
3.0 Overview 20 3.1 Application Development Process Model 20
3.1.1 Reuse-Based Model 21 3.2 Application Specifications 23
3.2.1 Hardware Requirements 23 3.2.2 Software Requirements 23
3.2.2.1 ARToolKitPlus S60 23 3.2.2.2 OpenGL ES 25 3.2.2.3 Carbide. c++ 26 3.2.2.4 S60 Platform SDKs for Symbian OS 26
3.3 System Requirements 27 3.4 Application Flow 28 3.5 Conclusion 28
Chapter 4- DESIGN AND DEVELOPMENT
4.0 Overview 29 4.1 Application Architecture 30 4.2 Application Development 31
4.2.1 Application Development's Obstacles 32 4.3 Coding Development 32
4.3.1 Map of'UNIMAS 32 4.3.2 The Building of Faculty of ('ognitive Sciences
and Human Development 34 4.3.3 Navigator Board 36 4.3.4 Keypad Functions 37 4.3.5 Rain FIlcct 39 4.3.6 Play Sound 40
4.4 Conclusion 40
Chapter 5- CONCLUSION
5.0 Overview 41 5.1 Discussion 41 5.2 Application Dchugging 44
5.2.1 Marker Select ion 44 5.2.2 Viewing from Ditlcrcnt Angles, Distances and
Lighting Condition 45 5.3 Strengths oFthe Application 46 5.4 Weaknesses of the Application 46 5.5 Future Recommendation 47 5.6 ('onclusion 48
V
REFERENCES 49
APPENDIX 53
vi
LIST OF FIGURES
Figure 1 The evolution and miniature of mobile AR: (a) Backpack with HMD, (b) UMPC, (c) handheld, (d) Mobile phone 2
Figure 2.1 Map Tracker 13
Figure 2.2 3D markers 13
Figure 2.3 AR-Tennis game 14
Figure 2.4 Visual Codes 14
Figure 2.5 A typical business card 15
Figure 2.6 A virtual avatar on a business card by using mobile-phone-based augmented reality 17
Figure 2.7 A 2D map, a 3D map and it photograph 18
Figure 2.8 Examples of'Mobile 3D maps 19
Figure 3.1 Reuse-Based Development Model 22
Figure 3.2 Level ofextended version for ARToolKit 24
Figure 3.3 Basic workflow of'an AR application using fiducial marker tracking 25
Figure 3.4 Flow of the application 28
Figure 4.1 Application architecture ol'Augm nted Business Card 30
vu
Figure 4.2 Code for texture loading
Figure 4.2.1 Code for texture rendering
Figure 4.2.2 Texture of UNIMAS map
Figure 4.3 Code for FCSHD building
Figure 4.3.1 Code for FCSHD building rotating
Figure 4.3.2 FCSHD textures (Top, Bottom, Front, Back, Lett and Right) and the 3D building
33
33
34
35
35
36
Figure 4.4 Code for Navigator Board 36
Figure 4.4.1 Texture of Navigator Board 37
Figure 4.5 Code fir Keypad Functions 38
Figure 4.5.1 Keypad and buttons ot'Nokia N95 38
Figure 4.6 Code tier rain ct1cct 39
Figure 4.6.1 Scrrcnshut of'raining rficct 40
Figure 4.7 Code iOr Sound etlret 40
Figure 5.1 Marker (business card) detection by mobile phone 42
Figure 5.2 Virtual neap and building displayed on mobile phone's screen 42
viii
Figure 5.3 Raining etlect with sound 43
Figure 5.4 Switch back to normal view of map by terminating the rain rendering 43
Figure 5.5 The "About" menu 43
Figure 5.6 Exit or quit from application 44
Figure 5.7 ARToolKit Marker Generator Online and the example values ofpixels in pattern file format (. pat) 45
Figure 5.8 Angle, distance and lighting condition that caused Mobile Phone (ARTooIKitPlus S60) unable to detect marker accurately 45
Ix
LIST OF TABLES
Table 1 Vary of standard size for Business Cards in several countries 16
Table 2 Hardware Requirements 23
x
ABSRTACT
MOBILE PHONE AUGMENTED REALITY BUSINESS CARD
Teo Tzong Ren
The main idea for this project is to introduce the technology of Mobile Phone Augmented Reality (AR) and develop it into an application for business card. AR technology is rather easier to introduce to the public by using Mobile Phone (Symbian OS) as compared to any other devices. Thus, this project bring up the basic idea of Mobile Phone AR technology and its application on a physical Business Card. Using a typical business card imprinted with a marker that assists in the marker tracking and location identification process, an AR 3D virtual map will be the information displayed on the Mobile Phone upon application execution. Furthermore, lower-end graphic is the challenge ofthis project in order to robust the capability of Mobile Phone tbllowed by enhances efficiency rendering process in real time with AR technology.
ri
ABSTRAK
MOBILE PHONE AUGMENTED REALITY BUSINESS CARD
Teo Tong Ren
7icjuun utunw projek ini uduluh untuk meny, erkenalkan tek, zulogi "Aagmcvned Reality" (AR) tek%n hi, nhit dan memhungunkun sun, aplikasi untuk kud
perniagaan. "1'eknologi AR lehih se. cuui dipe, kenulkan kepada musvurukut nelalui tele/inn hi, nhit (Svmhiun OS). Judi, pr(? jek ini menvtnnhung hiiuun asas tentctng AR tele%n himhit serta upliku. cinvu du/um Kad Perniuguun. 1)engan penggnnuun Kcrc! Pernicrguun yang hercetuk penundu . vertu menvunrhung dahlin proses penj(. jukcur tunda dun penenn, un lokasi. pela 31) , nuvu akan ditc{vung dulcnn
feldoll himhit . cehagui upuhi/u up/ikasi dijulankun. Se/ain inc, gru/ik
yang herkuuliti renduh mencpakun suluh sutu caharan da/am kujiun projek ini hugi nrc, itie. cuuiku, rnvu dengun teleJun himhit, sertu tuna ineningkijikan kelancurun proses. c meluki. c dulum muse nvuta dengun ne, >ggrncakun teknologi AR.
xii
CHAPTER I
INTRODUCTION
l. 0 Overview
This chapter discusses the background of the research, problem statements,
objectives, contribution, importance of research, scopes of the project, value and
significance of'the research as well as the structure of project.
1.1 Background
Augmented Reality (AR) technology, is a growing area that originated
from the Virtual Reality (VR) technology, yet vary from VR where the virtual
objects is superimposed upon or composite in the real world and the users are
interacting with the virtual objects in real-time (Vallino, I99S). AR require 6DOF
pose tracking ot'device such as head-mounted displays (IIMD), tangible intertiºce
object, etc, where the pose tracking must be inexpensive, work robustly and in
I
time through changing environmental conditions (Wagner & Schmalstieg, 2009).
However, the trend of AR devices had switched. This could be seen in figure I
from the first mobile augmented reality system - Backpack with I(MD (Figure
I a) that had been switched to a smaller and lower cost devices - Smart Phone
(Figure ld) (Rosenblum & Julier, 2009).
The term smart /phone was initially coined by unknown marketing
strategists to refer to a then-new class of cell phones that could facilitate data
access and processing with significant computing power (Zheng & Ni, 2006).
According toZhcng and Ni (2006). it smart phone is like it small, networked
computer in the firm ofa cell phone Which usually provides personal information
management (PIM) applications and some wireless communication capacity.
i 0000 -S,
4f iFt la) (b) (c) (d)
Figure 1: The evolution and miniature of mobile AR: (a) Backhack with I IMI),
(h) IJMPC. (c) handheld, (d) Mobile phone. (Wagner & Schmalstieg, 2009)
Wagner and Schmalstieg (2009) stated that smart phones are aiming liºr a
different market in AR compared to the more powerful and larger ultra mobile
PCs (UMPCs. Figure I b). Ile smart phones are surprisingly robust and R olproof
in applications development although its appearance is fragile. Achieving
sufficient performance fir AR applications are therefore require carefiºI choice of
algorithms and optimized code as smart phone has limited processing capabilities
compared to the PC platf6rm (Wagner & Schmalstieg, 2009). Most smart phones
come with built in camera which naturally lend itself to computer vision
approaches. The quality of AR in smart phone is therefore lower than the AR in a
PC platform as the computer vision's quality is based on camera and image sensor
characteristics (Wagner & Schmalstieg, 2009). Moreover, marketing has driven
the development of smart phone to more megapixels rather than higher video
quality (Wagner & Schmalstieg, 2009) which may increase the capability ofsmart
phone in perförming AR application.
In this project, the application of Mobile Phone AR in a typical business
card will he covered.
1.2 Problem Statement
Typical business cards come with aspect ratios of dinmensions range from
1.43 to 1.8. In United State and Canada, the size of a business card is 3.5 inch x
2.0 inch equivalent to the 1.75 aspect ratio (PrintingForLess. com, 2009).
These sizes of business card often includes the giver's name, company
affiliation (with logo) and contact information such as addresses, contact
number(s), e-mail addresses and website. 't'oday, a professional business card also
includes one or more aspects of'striking visual design such as map of'thc address
location.
The standard site of'a typical business card makes it impossible to allºICate
Inure Ill li/rlllatloll clue to the limitation of space results to the Ill lilrlllatloll included
in it business card is unclear or incomplete, hence creating confusion to its viewer.
An example of' such is the map attached in a business card that can sonletinles
confuse its viewer due to Ill furlllatloll of Its local loll Is unclear.
Hence, there is a need to find a way to solve such problem. With the task
at hand, this Project is to design and develop a business card application by using
Mobile Phone AR technology, fir enhancing the limctions of, traditional business
card. The idea of this application is to display clearer information through the
Mobile Phone AR technology within the space limitation in a physical object of
the real world.
1.3 Objective
The objective of the study is divided into two categories which is the
general objective and specific objLc: tives.
1.3.1 General Objective
The aim is to design and develop it Mobile Phone Augmented Reality
application for typical business cards.
1.3.2 Specific Objectives
The specific objectives of this project include:
" To design and develop 31) map using lower level graphics in
Svmbian OS Mobile Phone
" Implement 31) mar into it physical business card by using Mobile
Phone Augmented Reality technology.
1.4 Importance of Research
Despite the technology of Mobile Phone AR is still in an immature state.
the number oI smart phone usage. globally speaking is predicted t� rise up to KOO
million units by the year 20I2 (Rosenblum & Julier. ? OOH)). This trend tells that
the Mobile Phone AR technology is going to become it new broad area.
Applying the While Phone AR into a typical business card is beneficial
whereby it can save NO space and cost.. standard and typical site of a business
4
card has very limited space fbr displaying infbrmation, what ºmrre a map. This
problem can be solved by using a small tiny unique marker, which would be use
to help display the virtual object such as 3D map on the mobile phone screen later.
The research area does not only specify towards business card. The idea of
such application can also be applied into other materials or fields such as
magazines, leaflets, newspaper advertisements and etc.
1.5 Project Scope
The project only covers the enhancement of'business card by using Mobile
Phone AR technology. This is because the limitation exists where mobile phone
has lower capability in AR technology as compared to the PC platform. The scope
of the project is to digitize the use of typical business cards and adding more user
interface concept by using the Mobile Phone AR. This application can help in
increasing the effectiveness of information delivery through physical objects
(business card) to its viewer.
1.6 Values of Stud)
Although the use of mobile AR is currently in an unknown state, it is
believed that in time to come. it will he a powerful technology due to the device
that the technology would he run on is a smart phone (Wagner & Schmalstieg.
2009). The application that is designed in this project is an idea that can he use in
any area in the tiºture. The use of the application can save space and cost, and then
increase viewer's understanding. In addition. it shows the effectiveness use of
Mobile Phone AR ºn current mrld's uifbrmation delivery through a limited
physical space.
S
1.7 Significance of Study
The significance of the study is to expose Mobile Phone AR technology
and its application to the real world. The application introduces the use of Mobile
Phone AR technology in enhancing the function of typical business card by
merging user interface virtual 3D maps into the real environment.
1.8 Structure of Project
Chapter 1:
This chapter discusses the background of the research, problem statements,
objectives, contribution, importance of research, scopes of the project, value and
significance ofthe research as well as the structure of the project.
Chapter 2:
This chapter covers Augmented Reality (AR). Mobile Phone Augmented Reality,
Business cards and Mobile 31) maps. Apart from that, some examples of'existing
augmented business card will also be discussed. The end of'this chapter will then
cover the Mobile Phone AR in the application of'physical business card.
Chapter 3:
This chapter discusses about the application development process model,
application specifications tier K)th hardware and sollware which are going to he
used. Moreover, application flow will also he discussed.
Chapter 4:
In this chapter. the architectural design of the application developed is discussed.
More internal components of the application in the section of' application
architectural will he described.
6
Chapter 5:
The final chapter discusses the application validation, strengths and weaknesses of
the application, future recommendations and also the overall conclusion of the
project throughout the study.
1.9 Conclusion
This chapter introduced the overview of this project. The use of' Mobile
Phone AR technology is believed to he very well and much more accepted by the
public compared to PC. Besides, it is the easiest way to introduce AR technology
to the public as well. This project covers the design of business card application
fbr enhance its typical functions by Mobile Phone AR technology. The idea can
also he applied into other materials or fields like the newspaper, magazine, leaflet
and etc.
7
CHAPTER 2
LITERATURE REVIEW
2.0 Overview
This chapter covers Augmented Reality (AR), Mobile Phone Augmented
Reality, Business cards and Mobile 31) maps. Apart From that, some examples of
existing augmented business card will also he discussed. The end ot'this chapter
will then cover the Mobile Phone AR in the application of'physical business card.
2.1 Introduction of Augmented Rcalith
. augmented Reality (: AR) technology is it basic idea which superimposes
graphics, audio and other sense enhancements over real world environment in
real-time (Vallino. I99S). According to Wagner & Schmalstieg (200')). AR
require (i1)OF pose tracking device such as head-mounted displays (I IMI)).
R
tangible interface object, etc; where the pose tracking must be inexpensive, work
robustly and in real time through changing environmental conditions. AR
technology aims to enhance the user's perception and interaction with the real
world by implementing the real world with 3D virtual objects which appear to
coexist in the same space as the real world.
Vallino (1998) stated that when AR technology reaches the maturity level,
user should not ledl any conflict and discrepancies between the augmented
environment and the rules that user normally senses in the real world. Due to this,
the goal of AR technology is therelbre to create a system where user cannot
differentiate between the real world and the augmented virtual element. hence.
AR attempts to change those graphics to accommodate user's head and eyes
movements, so that the graphics would always lit the perspective.
Besides that, the need to know where the user is located in reference to his
or her surroundings is the biggest challenge faced by developers in AR research
(Bonsor, 2009). Another additional problem that exist in AR is tracking the
movement of users' eyes and heads. Bonsor (200k)) stated that a tracking system
has to recognize these movements and project the graphics related to the real-
world environment the user is seeing at any moment. Although such problem
exists, there are also some ways to increase the tracking accuracy. One of them is
using multiple GPS signals like military. An AR system with GPS receiver is able
to track user's position within an area precisely There ºs a system called real-time
kinematic (UPS being developed, where it can achieve centimetre-level accuracy
(Bonsor, 2009).
As Tong as researchers overcome the challenges that lice them, AR Will
likely penetrate every corner of our lives. The AR technology o1l6-s many
potential applications in various lields including maintenance and construction.
military, instant information as well as gaming. There are hundreds of' potential
applications for such it technology, gaming and entertainment being the most
obvious ones (Ronsor, 2009).
C)
2.2 Introduction of Mobile Phone Augmented Reality
In recent years, mobile phones had become an increasingly attractive
platform for augmented reality technology (Rosenblum & Julier, 2009). It is
predicted that the number of mobile phone will be sold in 2012 is 1.8 billion, and
800 of them are estimated to be smart phone (Rosenblum & Julier, 2009). The
term Smart Phone was first introduce by unknown marketing strategists to retcr to
a then-new class of cell phones which could facilitate data access and processing
with significant computing power (7_heng & Ni, 2006). Zheng and Ni (2006) also
stated that it smart phone is a small, networked computer in the form of .1 cell
phone which usually provides personal information management (PIM)
application and some wireless communication capacity.
Befbre the development of mobile phone AR, some research had been
clone in mobile AR to replace the cumbersome backpack plus head-mounted
display setups (Figure la) with ultra mobile P('s (tJMPC's, Figure Ih). This
evolution continues by the replacement oftJMP('s to PDAs (Figure lc) and Smart
Phone (Figure ld). According to Wagner and Schmalstieg (2009), smart phone
are aiming fir it dif1crent market in AR as compared to UMP('s. The smart
phone's AR performance is surprisingly robust and fbolproof although the
appearance ofthe devices is fragile.
Thus, in order to achieve the sufficient perliºrmance lr AR application,
choosing the right algorithms is important (Rosenhlum & Julien, 2009). This is
because smart phone has lower computing capabilities as compared to the P('
plattbrm Which it much higher end device. For instance, heavy use of template
CI+ code results in it prohibitive increase in code size (Rosenblum & Julien,
2009). Normally, most smart phone has built-in camera, and this allow the device
lends itself to the computer vision approach. Nevertheless, the quality ofcomputer
tracking is highly influenced by the camera and image sensor characteristics like
frame size. update rate, or lens distortion (Wagner & Schmalstieg, 2009).
I ()
2.2.1 Symbian S60
The S60 Platform is a software platform for mobile phones that runs on Symbian OS which is an operating system (OS) designed for mobile devices and
smart phones, with associated libraries, user interface, frameworks and reference implementation of common tools (Symbian Foundation, 2010). S60 is widely
used by most of the smart phones in the world whereby it was first created and
made open source by Nokia and contributed to the Symbian Foundation
(Symbian Foundation. 2010). According to Symbian Foundation (2010), the
Symbian occupied 47% of the smart phone platform market with 17.9 millions of handsets sold in the fourth quarter of year 2008 with an increasing figure every
year.
The reason why S60 platfbrm is preferred by most of the smart phones
company is that it consists of suite libraries and standard applications such as
telephony, PIM tools and Helix based multimedia layers (Symbian Foundation,
2010). In addition. S60 software is it multivendor standard for smart phones that
supports application development in ('+ +, Java MIUP, Python and Adobe Flash.
(Nokia. 2010).
There are ü ur major release of' S60 which are the "Series 60" (2001),
"Series 60 Second Edition" (2003), "Series 60 3"d Edition" (2005) and the most
current one. the "S60 5'r' Edition" (2008) (Nokia. 2009).
2.2.2 S60 Augmented Reality - 't'hc Map 't'rackin};
Langlot/. (2010) had creaicd it software fior Symhian phones that tracks
niaps, which are outfitted with regular grids of dots and tracked with 2,5 IX)F.
The map tracker performs six degree of freedom hose estimation from almost any
neap or other uniquely textured planar object. Small black dots are added into the
map to serve as it ret: rence point for tracking.
ii
According to Langlotz (2010), the map tracker makes use of Studiersuhe
Tracker's advance features such as the fast feature detection and poses estimation.
Also, the map tracker runs on Symbian at the frame rates of 15-30 fps for a
complete application. Besides this, the map tracker allows the map to be rotated
arbitrarily and tilting the phone to -45° as it treats maps planar as a full six degree
of freedom (Langlotz, 2010). The map tracker works by using the regular
Studierstube Tracker pipeline as a circle detector and a grid detector are adding in
it.
The steps which are perfbrmed fir every Frame by the map tracker is declared
by Langlotz (2010) as follows:
"1 hrc"sholdir>g The map trucker thresholds the image and utctomutirullv extracts dark
are as'.
" C'011 finu" %ullo titi»s,, The contour %ulluiwr searches for connected regions.
" Circle defection The c"irclc' clc'lc'clur checks all extracted cuiNuul: c (comlccYrcl regions)
using a simple rlli/zsr filling tilizurilhm.
(; r"id detection All circles are then handed over to the grid detector that tries to
recorrstricc"t the regcrlar grid of thc' clots nn the male. '1"he ,L
rid detector then
extracts the patches between the dots at a resolution of 05.05 pixels. This
high resolution for the hatches is required in order to track males with /rich frequencies (sma/l structures such as buildings or . streets at a si: e of
it /e'it mi/limeter". sl. which nould otherttict create aliasim. and hence result in bad detection quality.
" lOmhlate nuuclring The template matcher uses a(; au. c. cian downsamp/ing to reduce the patch size to .?
2. x3? pi. xelc and compares it to all halchr. c in the clataha. ce.
" /'o. cc' c'. ctilºIUholl
( oºl'c'c'Il1' c/c'rc'c'/c'c/ c'('//. c are u. cc'c1 for estimating the camera's 1ºo. cc rc'lcltitR' to the map resulting in a hl)OI"*/ºocc' tracking.
12
Figurc 2. I: Map Tracker by Langkotr (20I0).
2.2.3 History of Mobile Phone Augmented Reality
According to Wagner & Schmalstieg (2003), AR'I'oolKit has ported by
them to Window CE and created the first sell contained AR application on an oll-
the-shell embedded device. This port later evolved to become the AR'I'oolKitl'lus
library (Wagner & Schmalstieg. 2009).
In the year 2004. Mühring. Lessig. & I3imher created a tracker Ii)r mobile
phones that detects color-coded 31) marker shapes (Figure 2.1 ). 'I he system did
not take camera calibration or sub-pixel accuracy into account, thus causing its
accuracy to he very limited.
lb
i Figure 2.2: 31) markers by N16hring, l. rssig, & ßimbrr (2004).
13