A/Prof Manolya Kavakli - Macquarie Universitycomp.mq.edu.au › ~manolya ›...
Transcript of A/Prof Manolya Kavakli - Macquarie Universitycomp.mq.edu.au › ~manolya ›...
AUGMENTED
REALITY A/Prof Manolya Kavakli &
VISOR team
Majed Alrowaily
Shaghayegh Asgari
Guillaume Lucas
Farzana Alibay
Hessam Jahani
Department of Computing
Macquarie University
Sydney, Australia
Sydney Science Festival 2016 13th August 2016
Associated Research Groups
VISOR (Virtual and Interactive Simulations of Reality)
CEPET (Centre for Elite Performance, Expertise, and Training)
Understanding expertise is important for the design of training programs.
Principles and mechanisms proposed to underlie expertise can be used to evaluate the theories about basic cognitive processes and capacities, and thus explain human performance more generally (Loft et al, 2009).
CAVE @ ENSAM, Chalon Sur Saone, France
Simulation Hub & VR Lab 2015
Simulators
VISOR @ VR Lab
8 Postdocs
12 PhD students graduated
8 current PhD students
8 to start next term/year
7 Honours students
25 MIT students
11 MEng interns
6 Programmers
85 in total
VISOR: Virtual and Interactive Simulations of Reality
HCI
Virtual Reality
Augmented Reality
Graphics Programming
Simulation Design
Motion Tracking
Training & Expertise
Scene Complexity
Simulator Sickness
Novices and Experts
Performance
Cognitive Processing
Cognitive Load
Human Information Processing
Joint performance of Human Computer/ Machine Interfaces
Augmented Reality
A type of virtual reality in which
additional information,
otherwise imperceptible to the human system
is made perceptible and
registered with the display of the physical world.
Reality-Virtuality Continuum
Milgram (Milgram and Kishino 1994; Milgram, Takemura et al. 1994) describes a taxonomy that identifies how augmented reality and virtual reality work are related.
The real world and a totally virtual environment are at the two ends of this continuum with the middle region called Mixed Reality.
Augmented reality lies near the real world end of the line with the predominate perception being the real world augmented by computer generated data.
Augmented virtuality is a term created by Milgram to identify systems which are mostly synthetic with some real world imagery added such as texture mapping video onto virtual objects. This is a distinction that will fade as the technology improves and the virtual elements in the scene become less distinguishable from the real ones.
Augmented Reality
a live, copy, view of
a physical, real-world environment whose
elements are augmented by computer-generated
sensory input such as sound, video, graphics or
GPS data
Augmented Reality
Imagine a VR head mounted display,
but the display doesn't block out the regular view, it's just superimposed on it. Imagine walking around a building and "seeing" inside
the walls to the wiring, plumbing, and structure.
Or, seeing the tumor inside a patient's head as you hack away at it.
Focuses on enriching the natural environment
Extra information is displayed on a HMD Currently mainly used in the military with few civilian
applications
In the Touring Machine scenario,
the MARS unit acts as a campus information system, assisting a user in finding places and allowing her to query information about items of interest, like buildings, statues, etc.
MARS: Mobile Augmented Reality Systems
Mobile market
Rapid changes since 2007 in
Single purpose handheld devices:
Apple iPhone and
Android systems (Samsung’s Galaxy)
Tablet devices
(integrating smartphones and netbooks):
Apple iPad
Android systems (Samsung’s Galaxy)
The Layar Reality Browser An augmented reality app
(mobile phone application) that
shows you what is around you by displaying real time digital information (layers) on top of reality.
This digital information is called a ‘layer’.
Layers can provide services, such as finding ATMs, houses for sale and restaurants including reviews.
Layers can also provide an experience with interactivity, 3D objects and sounds for games and engaging guided tours. Many layers provide both.
AR platform? Essentially you’re using your phone like a camera
so you can see the real world as you pan around,
but the application utilizes your GPS location and G-Sensor
to determine exactly what your phone/camera is pointing towards and
provides you more information about it directly on your phone!
But Layar technology isn’t just an application…
its a platform.
The Layar Reality Browser first launched in June 2009 in The Netherlands,
announced that it’s second generation Reality Browser is now available globally on Android devices.
Many new content layers are available in the Reality Browser which vary from
Wikipedia, Twitter and Brightkite to local services like Yelp, Trulia, store locator’s, nearby bus stops, mobile coupons, Mazda dealers and tourist, nature and cultural guides.
http://www.youtube.com/watch?src_vid=b64_16K2e08&v=HW9gU_4AUCA&feature=iv&annotation_id=annotation_121798
http://www.youtube.com/watch?v=b64_16K2e08
http://layar.pbworks.com/w/page/7783224/Creating%20the%203D%20objects
AR games
In 2010:
To further grow the community of developers, Layar has released an additional 500 new API keys.
The supporting wiki (http://layar.pbworks.com),
with developer instructions and documentation, are all now available.
Augmented Reality Applications
Imagine that:
Signs could broadcast their message in several languages, being automatically picked up and displayed on the users phone in the appropriate language.
A map with GPS and/or building information could be projected on to the phone –> No more getting lost.
Virtual tour guides could be downloaded for any country and location.
Multimodal support for visual-impaired, using brail…
The possibilities are endless…
Using the GPS location, accelerometer and gyroscope of the smart tablet, and Google glasses, we can generate a mobile AR system.
The AR system (I-DeSIGN) will facilitate design communication by
using 3D architectural objects such as walls and windows
to push and pull to shape and create a virtual built environment,
whilst the architect has the benefit of having a superimposed image of the virtual world in physical reality.
iDesign A Multimodal Augmented Reality System for Spatial Design
Building Information Modeling
Simulation Hub
3D visualisation in Augmented Reality
Benefits of Augmented Reality
offers a combination of physical and virtual objects.
aids learning due to the combination of its relevance to the learner’s physical world and
the customizability of the simulation to offer a virtual world.
Therefore, the learning experience is superior to other media such as computer games and virtual reality because
does not separate the users from the reality but, instead, realistically transforms their reality.
This transformation is represented by a temporary scaffold between
what is real and virtual, in other words, known and unknown.
controls the accessibility, transparency and visibility of this scaffold as learning progresses.
AR frameworks & Scaffolding discuss learning from mainly three perspectives:
physical,
cognitive, and
contextual.
On the physical dimension, they argue that physical manipulation affords natural interactions and
encourages the creation of embodied representations to support learning.
On the cognitive dimension, they discuss how spatiotemporal alignment of information through AR systems can aid
student’s symbolic understanding by scaffolding the progression of learning, and result in improved understanding of abstract concepts.
On the contextual dimension, they argue that AR creates possibilities for interactive learning,
ultimately facilitating personal association and personally meaningful experiences.
VR & AR http://www.youtube.com/watch?v=bBjvqnKQsTI&list=P
LDF1BBECCE066EE5E
https://www.youtube.com/watch?v=V34gCw4fyLs
A standard VR system
A standard virtual reality system seeks to completely immerse the user in a computer generated environment.
This environment is maintained by the system in a frame of reference registered with the computer graphics system that creates the rendering of the virtual world. For this immersion to be effective, the egocentered frame of reference
maintained by the user's body and brain must be registered with the virtual world reference.
This requires that motions or changes made by the user will result in the appropriate changes in the perceived virtual world. Because the user is looking at a virtual world there is no natural connection
between these two reference frames and
a connection must be created.
(Azuma 1993).
A standard AR system An augmented reality system could be considered the ultimate
immersive system. The user can not become more immersed in the real world.
The task is to register the virtual frame of reference with what the user is seeing.
This registration is more critical in an augmented reality system because we are more sensitive to visual misalignments than to the type of vision-kinesthetic errors that might result in a standard virtual reality system.
The combination of real and virtual images into a single image presents technical challenges for designers of augmented reality systems. How to do this merging of the two images is a basic decision the designer must make.
Milgram and Kishino 1994; Milgram, Takemura et al. 1994 categorize augmented reality systems using Extent of Presence Metaphor that directly relates to the display that is used.
the multiple reference frames that must be related in an augmented reality system.
Display based AR At one end of the spectrum is monitor based viewing of the
augmented scene. This has sometimes been referred to as "Window on the World" (Feiner, MacIntyre et al. 1993) or Fish Tank virtual reality (Ware, Arthur et al. 1993).
The user has little feeling of being immersed in the environment created by the display. This technology is the simplest available. It is the technology that several other systems in the literature uses (Drascic, Grodski et al. 1993; Ahlers, Breen et al. 1994).
HMD based AR
To increase the sense of presence Head-mounted displays (HMD) have been widely used in virtual reality systems.
Augmented reality researchers have been working with two types of HMD: video see-through and
optical see-through.
The "see-through" designation comes from the need for the user to be able to see the real world view that is immediately in front of him even when wearing the HMD.
Video see-through AR The standard HMD used in virtual reality work gives the user complete
visual isolation from the surrounding environment.
Since the display is visually isolating the system must use video cameras that are aligned with the display to obtain the view of the real world.
This can be seen to actually be the same architecture as the monitor based display described above except that now the user has a heightened sense of immersion in the display.
Optical see-through AR The optical see-through HMD (Manhart, Malcolm et al. 1993) eliminates
the video channel that is looking at the real scene.
Instead, the merging of real world and virtual augmentation is done optically in front of the user. This technology is similar to heads up displays (HUD) that commonly appear in
military airplane cockpits and recently some experimental automobiles.
In this case, the optical merging of the two images is done on the head mounted display, rather than the cockpit window or auto windshield, prompting the nickname of HUD on a head.
See for calibration: http://stctutorial.icg.tugraz.at/
Main Advantage of Augmented Feedback
the ability to overlay virtual information on reality.
AR systems are successful because
they display information relevant to the user
at the appropriate time and location.
Most AR systems leverage this spatiotemporal contiguity by overlaying virtual information relevant to physical objects and spaces.
Thus, AR technology can bridge the gap between
physical manipulatives and their symbolic representation
by morphing the physical object into its representation.
Advantages of AR Feedback Physical
spatial and temporal alignment,
mapping between abstract and physical manipulatives,
Cognitive
situated cognition,
instructional scaffolding,
Contextual
engagement through personalisation,
motivation through emotional connection.
Disdvantages of AR Feedback
Cognitive load
Reality blur
blurring between borders of reality and virtual,
Dependency on multi-sensory information and stimulation
which refers to addiction to feedback and
Accessibility of AR technology.
Advantages and Disadvantages of AR Displays
discussed in greater detail by Azuma (Azuma 1995).
There are some performance issues: With both of the displays that use a video camera to view the real world there is a
forced delay of up to one frame time to perform the video merging operation.
At standard frame rates that will be potentially a 33.33 millisecond delay in the view seen by the user.
Since everything the user sees is under system control compensation for this delay could be made by correctly timing the other paths in the system.
Or, alternatively, if other paths are slower then the video of the real scene could be delayed.
With an optical see-through display the view of the real world is instantaneous so it is not possible to compensate for system delays in other areas.
On the other hand, with monitor based and video see-through displays a video camera is viewing the real scene. An advantage of this is that the image generated by the camera is available to the system to provide tracking information.
The optical see-through display does not have this additional information. The only position information available with that display is what can be provided by position sensors on the head mounted display itself.
How to develop AR apps
Wikitude forecast that AR/VR will hit $120 billion in revenue by 2020, with AR taking the lion’s share at around $90 billion and VR at $30 billion.
AR SDK for Google Glass, Epson Moverio, Vuzix M100 and ODG R-7. The Wikitude SDK is available for both Android and iOS devices.
Wikitude SDK + Cordova
Wikitude SDK offers a plugin for Unity3D so
you can integrate Wikitude’s computer vision engine into a game or application fully based on Unity3D.
Wikitude SDK works with up to 1000 images that can be recognized offline.
Apache Cordova is an open-source mobile development framework.
It allows you to use standard web technologies - HTML5, CSS3, and JavaScript.
Vuforia SDK Vuforia is an AR Software Development Kit (SDK) for mobile devices.
It uses Computer Vision technology to recognize and track planar images (Image Targets) and simple 3D objects, such as boxes, in real-time. This image registration capability enables developers to position and orient virtual
objects, such as 3D models and other media, in relation to real world images when these are viewed through the camera of a mobile device.
The virtual object then tracks the position and orientation of the image in real-time so that the viewer’s perspective on the object corresponds with their perspective on the Image Target, so that it appears that the virtual object is a part of the real world scene.
The Vuforia SDK supports a variety of 2D and 3D target types including ‘markerless’ Image Targets, 3D Multi-Target configurations, and a form of addressable Fiduciary Marker known as a Frame Marker.
Additional features of the SDK include localized Occlusion Detection using ‘Virtual Buttons’, runtime image target selection, and the ability to create and reconfigure target sets programmatically at runtime.
provides Application Programming Interfaces (API) in C++, Java, Objective-C, and the .Net languages through an extension to the Unity game engine.
supports both native development for iOS and Android
Using Unity Game Engine
One can introduce a plan as an image target to Unity.
Then, download the 3D form of buildings from google earth.
Afterwards, with one of compatible converter we need to convert 3D models to .FBX file.
As long as you have .FBX file, in Unity,
you can match it with the plan to show the building when the camera detects the plan in environment.
How to develop AR apps
**Setup the development environment:
1- Installing Cordova Cordova command-line runs on Node.js (https://nodejs.org) and
is available on NPM. Follow platform specific guides to install additional platform dependencies.
Open a command prompt or Terminal, and type npm install -g cordova.
2- Installing Android SDK http://developer.android.com/sdk/installing/index.html?pkg=tools
3- Downloading Wikitude sdk and Get your trail license http://www.wikitude.com/products/extensions/cordova-plugin-augmented-
reality/
http://www.wikitude.com/developer/licenses
**Setup Cordova application:
1- Create a project cordova create MyApp
2- Add a platform cd MyApp
cordova platform add android
3- Add Wikitude SDK Plugin
cordova plugin add $path-of-sdk
4- Add your wikitude license key at WikitudePlugin.js this._sdkKey = "Your-License-key";
5- Create WTC file for your target Image using Wikitude Target manager web application https://cas.wikitude.com/cas/login?service=https%3A%2F%2Ftargetmanag
er.wikitude.com%2F
6- Create Your AR world "See example application"
7- Load your AR world "See example application"
For more information see wikitude documentation http://www.wikitude.com/developer/documentation/phonegap
http://developer.android.com/tools/device.html
http://www.comp.mq.edu.au/~manolya/
We are all looking for an answer but in fact what drives us is the question. Future isn’t written. It is designed.
Using Psycho-physiological feedback in AR
The HCI technologies that we study today will provide the interfaces of tomorrow.