hikar: augmented reality for walkers – latest developments

Nick WhiteleggMaritime and Technology FacultySouthampton Solent University

Augmented realityAugmented reality (AR): the augmentation of the real-world (typically, a mobile device's camera feed) with electronic data

Applications of AR include:

Detecting real-world objects in the camera feed and overlaying with computer models (e.g AR chess [1])

Geographical applications in which GPS and compass used to overlay geodata on camera feed

Work so far mostly on overlaying POIs (e.g [2]); however Hikar attempts to overlay OSM ways for navigation

What hikar does

Overlays OSM highways on camera feed

Intended to be usable for in-the-field countryside navigation

Initial work presented at SOTM 2011

This talk will revise the details of the project and summarise further work and experiments

With demo videos!

Summary of recent developments

Upgraded phone from HTC Hero to Galaxy SIII

Important hardware usability consequences

Rewritten app for Android 4.0+ rather than 2.1+

Useful new features such as the ability to use the camera feed as an OpenGL texture

Previously, there was no easy way to combine the camera feed and OpenGL: original approach worked but “wasn't supposed to”

Now uses OpenGL ES 2.0 rather than 1.0

Initial work on line-of-sight

AR for countryside navigation

Footpath data from OpenStreetMap is overlaid on the camera feed

This could be used in situations such as:

You enter a field, and it is unclear where the exit is

You are at a junction of paths and it is unclear which is the official footpath

You need to find your way across a moor with no apparent path

hikar and AR: how does it work?

Components:

Camera feed from phone

OpenStreetMap data [3] overlaid on feed – delivered as 5km x 5km XML tiles from Freemap servers

Freely available height data (Ordnance Survey Land-Form PANORAMA) [4] used to adjust OSM data for height on the client

Height-adjusted OSM data cached on device for next time

Height data

Height data must be accounted for

If there's a hill in the way, it would be desirable to overlay the OSM data on the slope of the hill, rather than “burrowing into the hill”...

How is the AR done?We need to know our current position and the device orientation

Obtain GPS position using the Android Location APIs[6]

Use the Android Sensor API [7] to obtain the orientation of the device (bearing and pitch) expressed as a rotation matrix

This matrix represents the rotation from the phone in standard orientation to the actual orientation

Phone orientation

Role of OpenGLThe standard cross-platform 3D API, OpenGL [10] is used to render the map data on the device (specifically OpenGL ES 2.0)

OpenGL defines a "modelview matrix" [11] which describes the relation between the 3D world and our current view of it

The modelview matrix translates world coordinates (absolute position in the world) to eye coordinates (our current view of them: origin centre of screen, z increases outwards)

The matrix returned from the sensor API – remapped if necessary - can thus be used as the modelview matrix

Incorporating the camera feed

The camera feed is loaded as an OpenGL texture [12]

A rectangle is drawn on whole screen before drawing the map data

This rectangle is textured using the camera feed

SurfaceTexture

Since Android 3.0 the SurfaceTexture class [13] allows camera feeds to be used as an OpenGL Texture

Create a SurfaceTexture using an OpenGL texture ID (which is created in the normal way for textures, i.e glGenTextures())

Issues - thenIn 2011 and testing on the HTC Hero the main

problems were:

Hacked approach, not recommended by Google, to combining OpenGL and camera feed (multiple SurfaceViews not supposed to be overlaid)

Instability of compass, despite exponential smoothing

GPS and OSM data inaccuracies

Now...Much faster and more stable on the Galaxy SIII

Compass accuracy much better, little of the “swaying around” seen under the Hero

GPS accuracy and OSM data accuracy still an issue

Ways generally head the right direction (indicating good compass accuracy) but are often slightly displaced to left or right of where they should be (see video clip)

Remaining issuesGPS and OSM data accuracy (see last slide)

Notably true in urban areas (urban canyon effect) or in woods, predictably

Need to hold device steady, otherwise the overlay “floats around” somewhat

Do people want to do this?

Issues with quality of camera feed can mean it is difficult to locate reference objects (e.g. trees) when working out course of a path

Unreliable with large amounts of data (Birmingham!)

UsabilityIn tests, the app has been useful to work out which direction, in general, to head across a field

Also useful for finding location of a turn-off

Reasonably accurately shows course of path up a hill

GPS/OSM data accuracy issues means that it is less useful for working out which side of a fence, tree, etc a path goes

A few metres might not mean a lot on a map, but does here!

Line of sight

One issue is that footpaths behind hills need to be hidden

Code to do this has been implemented in the source code but not turned on yet in the .apk (insufficient time for testing)

Line of sight – how it worksFor each way, a line is drawn between it and the current GPS location

The DEM grid squares passed over are calculated using Bresenham's line drawing algorithm [16]

For each grid square, the actual height on the DEM, and expected height assuming a uniform slope from GPS location to way, are compared

If the actual height of any square is higher than the expected, there is an obstruction and the way is (or certain points within the way are) invisible

Line of sight

Downloading hikar

Currently a research project rather than a user-orientated app

… but nonetheless usable for some situations

Currently uses OSGB projection, so only suitable for UK in its current state

APK (health warning!) available atwww.free-map.org.uk/downloads/android/hikar040913.apk

Source (GPL) on githubhttps://github.com/nickw1/Freemap/blob/master/java/hikar/src/

freemap/hikar/

AvailabilityHikar restricted to certain counties in England due to availability of data from Freemap server

SOUTH: Hampshire, West Sussex, Surrey, Wiltshire, Somerset, Buckinghamshire

NORTH: Cheshire, Derbyshire, Lancashire, Cumbria, North Yorkshire

MIDLANDS: Oxfordshire

However, if the data is on the server and the OSGB projection is replaced with another, it could potentially be used anywhere

The future – Google Glass?

The arrival of Google Glass may open up exciting new possibilities for augmented reality

If you can put up with wearing a Google Glass while out walking, hikar could be a good navigation aid

The issues with holding the device steady would be less of a problem here, though GPS accuracy would remain an issue

ConclusionsConcept remains promising, albeit still with some issues

Significantly more usable on a contemporary, high-spec phone than an early Android device

Issues largely concern accuracy of OSM data and GPS, and the need to hold the device steady

Improved GPS accuracy in particular would improve usability

hikar and Google Glass open up some interesting possibilities!

References[1] New Scientist (2011), “Augmented reality teaches you chess”, online, available

http://www.newscientist.com/blogs/nstv/2011/02/augmented-reality-system-teaches-you-chess.html, last accessed 30/6/13

[2] Wikitude (n.d) "Wikitude" (online) Available: http://www.wikitude.com/en/, last accessed 30/6/13

[3] OpenStreetMap (2013), “OpenStreetMap”, (online) available http://www.openstreetmap.org, last accessed 30/6/13

[4] Ordnance Survey (2011) "Land-Form PANORAMA - small-scale height data of Great Britain",(online) available http://www.ordnancesurvey.co.uk/oswebsite/products/land-form-panorama/index.htmlast accessed 30/6/13

[5] NASA (2009) "Shuttle Radar Topography Mission", (online) Available: http://www2.jpl.nasa.gov/srtm/, last accessed 30/6/13

[6] Google (2013) "Location and Maps" (online), Available: http://developer.android.com/guide/topics/location/index.html, last accessed 30/6/13

[7] Google (2013) "SensorManager" (online), Available: http://developer.android.com/reference/android/hardware/SensorManager.html, last accessed 30/6/13

[8] Haseman, C. (2009) “Augmented Reality on Android: Using GPS and the Accelerometer” (online), Available: http://www.devx.com/wireless/Article/43005/0/page/2, last accessed 30/6/13

[9] Andreasson, M. (2010) “Technical: Orientation sensing in Photos Around” (online), Available: http://photosaround.mandreasson.com/2010/05/technical-orientation-sensing-in-photos.html, last accessed 30/6/13

[10] Khronos Group (1997) "OpenGL - The Industry Standard for High Performance Graphics" (online), Available: http://www.opengl.org/, last accessed 30/6/13

[11] Neider, J., Davis, T. and Woo, M. (1994) "OpenGL Programming Guide" Chapter 3, (online) available: http://fly.cc.fer.hr/~unreal/theredbook/chapter03.html, last accessed 30/6/13

[12] Neider, J., Davis, T. and Woo, M. (1994) "OpenGL Programming Guide" Chapter 9, (online) available: http:://fly.cc.fer.hr/~unreal/theredbook/chapter09.html

[13] Google (2013) “SurfaceTexture” (online) Available: http://developer.android.com/reference/android/graphics/SurfaceTexture.html last accessed 30/6/13

[14] Google (2013) “Camera” (online) Available: http://developer.android.com/reference/android/hardware/Camera.htmlllast accessed 30/6/13

[15] Google (2013) “Camera.Parameters” (online), Available: http://developer.android.com/reference/android/hardware/Camera.Parameters.html, last accessed 30/6/13

[16] De Beauclair Seizas, R., Riguette Mediano, M., Gattass, M. (1999) “Efficient Line-of-Sight Algorithms for Real Terrain Data”, online, availlable: http://www.tecgraf.puc-rio.br/publications/artigo_1999_efficient_lineofsight_algorithms.pdf, last accessed 30/6/13

Thanks!

Any questions?

nick_whitelegg@yahoo.co.uk

freemapinfo@gmail.com