Head Mounted DisplaysSimon Chuptys

KU LeuvenLeuven, Belgium

simon.chuptys@student.kuleuven.be

Jeroen De ConinckKU Leuven

Leuven, Belgiumjeroen.deconinck@student.kuleuven.be

ABSTRACTHead Mounted Displays (HMD) are being heavily researchedagain, both as a new display technology and interactionmedium. While they have been losing popularity after theirinitial hype, recent technological developments have stimu-lated the interest for HMD research again. In this paper wegive an overview of the research that has been done relatedto head mounted displays. We will describe the applicationsof HMDs along with the major difficulties that are associatedwith them.

Author Keywordshead mounted display; augmented reality; virtual reality;screen;

INTRODUCTIONHead mounted displays have quite a long history of research.Initial designs, like the sword of Damocles by I. Sutherland(figure 1) or the Virtual Boy by Nintendo (figure 2) were builtas heavy helmets strapped to the user’s head, making themuncomfortable to wear for prolonged periods of time whilealso severely restricting head movement (let alone full bodymovement).

Figure 1. The ’sword of Damocles’ by I. Sutherland (1968). Consideredto be the first head mounted display.

Recent technological advancements have made the HMDlighter and smaller, in some cases barely different from nor-mal glasses. Due to these advancements, HMD technologyhas become an exciting area of research once again. There-fore, we believe it is very important to provide an overviewof current state of research, the problems typically associatedwith HMDs, the products that are currently available or indevelopment and the application areas that may benefit fromthis technology.

HEAD MOUNTED DISPLAYSBasically, head mounted displays include every kind of tech-nology that mounts displays to the user’s head. We can ob-serve two major areas that make use of HMDs: AugmentedReality (AR) and Virtual Reality (VR). While they both usesimilar devices, they each have very specific application ar-eas and problems associated. Therefore, we think it feasibleto address them separately.

Figure 2. The ’Virtual Boy’ by Nintendo (1995)

Augmented RealityThe idea behind Augmented Reality (AR) is to enhance thenormal vision by means of overlaying images using displaysclose to the eyes. This way, the wearer is assisted in every-day tasks. Possible applications include displaying relevantinformation about current activities, highlighting importantenvironmental elements, or feeding the wearer information ina way that he can still carry on with normal activities.

There are different possible configurations for using HMDsin AR: both a monocular (only one eye has a display) or abinocular setup can be used. Furthermore, the displays maybe either transparent or opaque. Overlaying images onto thenormal vision can be done in two ways: either using videosee-through (VST) displays or optical see-through (OST) dis-plays.

VST-HMDs capture a video stream of the environment usingcameras, which then is mixed with artificial content. The re-sulting images are typically projected on an opaque screen,so the user will only see images that are captured by the cam-eras mounted on the HMD. The advantage of this method isthat conventional computer vision techniques can be used tosynchronize the real-word and artificial imagery. Disadvan-tages include a limitation on the field of view (fov), as wellas a lessened ability to focus on specific objects that are atdifferent distances from the user. Also, latency issues mayarise.

1

OST-HMDs on the other hand make use of see-through dis-plays, allowing the user to see real-world imagery in the con-ventional way, without fov, focussing or latency issues (atleast, not for the real-world images). These devices typi-cally make use of a half-silvered mirror to project the digi-tal images onto while not obstructing the normal view (seefigure 3). Virtual images from a screen are reflected (di-rectly or by an intermediate reflection on a curved mirror)on a half-silvered mirror. While reflecting the virtual images,the half-silvered mirror allows more or less unobstructed nor-mal viewing. The disadvantages of this method are mostlydue to the ever changing environmental conditions: changesin lighting call for careful real-time adjustments in brightnessand contrast. Furthermore, these AR devices typically suf-fer from visual interference, due to the overlaying of multi-ple image sources. Calibrating virtual images with what theuser sees is also non-trivial. However, the OST method hasbeen shown to offer the most natural viewing conditions [1].Therefore, this is the most promising technique to considerregarding AR.

Figure 3. Schematic overview of a typical OST device. Virtual images(blue) are reflected on a half-silvered mirror while normal viewing (grey)remains unobstructed. Image by D. Jackel.

The popularity of AR has dramatically increased these pastfew years, mainly because of the development of GoogleGlass [2], which will allow accessibility to this technologyfor the general public.

Virtual RealityVirtual Reality (VR) is where the HMD blocks out our nor-mal vision and replaces it with a rendering of a virtual world.This is generally associated with entertainment purposes suchas immersion in video games, but there are a lot of situa-tions that can benefit from VR. Other example uses of VR arearchitectural visualisation, teleconferencing and visualisationof molecular structures.

Replacing the entire visual field with an image on a displaycomes with a whole set of problems, mostly related to the dif-ferences between viewing the continuous light coming fromthe real world and viewing the discretized light emitted by anHMD. Like AR, VR is currently gaining popularity due to aHMD for the general public being in development (the Riftby Oculus VR [3]).

Figure 4. A developer model Oculus Rift (source: oculusvr.com)

PROBLEM DESCRIPTIONFollowing table gives an overview of the main problems re-lated to head mounted displays. For each problem, the tablelists if the different subtypes of HMD (VST, OST, VR) areaffected by it (+) or not (-). The remainder of this sectiondescribes each of these problems in more detail.

AR (OST) AR (VST) VREnvironmental conditions + + -Latency - + +Registration accuracy + - -Binocular rivalry (1) (1) -Depth of focus + + +Visual displacement - + +Relative eye movement (2) + +

(1) when using a monocular setup(2) application-specific (eg tracking of real world objects)

Augmented RealityUncontrollable conditions: Due to the ever changing light-ing conditions, AR devices continuously need to make adjust-ments to the brightness and contrast of overlay images [4].VST-HMDs have the specific problem of a limited dynamicrange due to the use of cameras to capture real world imagery:over-bright or very dark real world footage may cause thevideo stream to become unrecognisable, literally blinding theuser. Modern displays still can’t display the dynamic range ofreal world luminance. OST-HDMs suffer from another prob-lem, which is amplified by lack of proper occlusion betweenreal and virtual images. This means that it is impossible tovisualize ’black’ overlays: these regions appear transparent.This decreases readability of the overlaid footage in outdoorenvironments.

Latency, resolution, display curvature: As mentioned be-fore, VST-HMDs severely restrict the user’s field of view.Kollenberg shows that the reason is twofold: foveal vision(vision inside the center of gaze, see figure 5) can be restrictedby the display resolution, while peripheral vision (vision out-side the center of gaze) may be blurred by the display curva-ture [5]. Furthermore, the delay due to capturing real worldimages and displaying them on the HMD introduces a slightlatency during eye and head movement. This results in slowerreaction times and reduced hand-eye coordination abilities ofthe wearer.

Registration accuracy: When overlay footage needs to in-teract with the real world imagery, we need good calibrationtechniques in order to obtain a high registration accuracy.

2

Figure 5. Schematic overview of the eye indicating foveal and peripheralvision.

VST-HMDs typically suffer less from these issues, since alot of the techniques of computer vision can be used to mixreal world and virtual footage. OST-HMDs however typicallycan’t use these techniques because of the lack of recordingsfrom the real-world footage as seen by the user: The compo-sition of real and virtual images happens on the user’s retinainstead of on the display. However, in [1] a method is ex-plained for calibrating these OST devices. An extra videocapture stream is used for this process, allowing to still usecommon computer vision techniques while calibration errorsdue to the facial structure of the wearer are minimized. Evi-dently, by using this technique the latency problem is reintro-duced.

Binocular rivalry: At each point in time, there is usuallyone eye which is more dominant than the other, meaning wemainly see images seen by that eye, while parts of the im-agery captured by the other eye is discarded. Which eye isdominant and the duration of this dominance varies and isinfluenced by what is currently observed, making it unpre-dictable. This causes problems which are mainly apparentwhen using monocular HMDs because each eye sees a differ-ent image in this case: depending on the what the wearer islooking at, the eye with the virtual footage may not be domi-nant, resulting in the inability to see the virtual imagery.

Depth of focus: In normal viewing, our eyes adjust them-selves based on the distance to the object we are focussingat. When VST techniques are used however, all imagery re-sides at the same distance from the eyes, and only one focaldistance is provided by the camera capturing the footage. Sothe perceived focal distance differs from the focal distanceresulting from conventional viewing. When using monocularVST devices, the problem is even worse and could lead to oneof two eyes being out of focus. For OST devices, the prob-lems arise due to the varying focus of the real-world imagery(perceived by the wearer’s eyes in the conventional manner)while focal distance of the digital images remains constant.[7] gives an overview of potential fixes to this problem, oneof which makes use of a mechanism allowing dynamic ad-justments of the focal distance (see figure 6). The methoduses an adjustable liquid lens, placed between the eye and thedisplay, which is able to translate the focal distance from in-finity to the near view point of the eye. A major advantage

of this approach is that it can be used in almost all existingconfigurations.

Visual Displacement: The recorded view shown on a HMDmay not match the view the user is used to. The cameras ona VST-HMD may be offset from the expected position (usu-ally they are placed slightly more to the front and top than theuser’s actual eyes) and the distance between the viewpointsfor the left and right eye of the user may not match their ac-tual inter pupillary distance. The former affects visuomotorperformance and can be compensated for by adaptation if thedisplacement is not too extreme. The latter may cause eyestrain.[10]

Figure 6. Schematic overview of an OST device allowing dynamic read-justment of focal distance. Image by D. Jackel.

Virtual RealityDepth of focus: As is the case with AR, the perceived dis-tance of the screen as sensed from eye accomodation is fixedin a HMD, our eyes have to focus their lenses on this depth tosee a sharp image. However, the distance information we re-ceive from the HMD through stereo disparity does not matchthis (and can not match this since it varies for different pointson the screen and different points in time). This mismatch ofsensory information can cause eye strain and headaches. Theonly solution to this problem so far is to wait for our brain toget used to it.[9][8]

Latency: The computer rendering the virtual world displayedon the HMD needs some time to calculate the image fromthe correct viewpoint every time we move our head, and thedisplay updates at a fixed interval which can cause delays aswell. Because of this, there will always be some degree oflatency to updating the screen in reaction to our actions. Thiscan be perceived as the whole world moving slightly, but theviewer can not feel this movement in their vestibular system.The reaction to mismatched vestibular inputs is motion sick-ness. Because of this, it is essential to keep latency as lowas possible. This is done by using displays with high framerates and lowering the quality of the rendering of the virtualworld in order to achieve these higher frame rates . Loweringlatency thus negatively impacts the aesthetic quality and theprice of the HMD.[8][13]

3

Eyes moving relatively to the screen: The The Vestibulo-Ocular Reflex rotates the eyes in response to head movementsin such a way that we keep looking in the same direction andthat the image we see stays stable. Though as discussed inthe previous paragraph on latency, the display doesn’t updateimmediately in reaction to head movements. So the displaymoves with our head while our eyes keep looking in the samedirection, and the image displayed on the screen will look asif it is constantly shifting away for a bit then snapping backinto position. This same effect will also occur when trackingan object on the screen. This is explained in image 7 wherethe position of an object relative to the eye is plotted on thehorizontal axis and time on the vertical. In the top left imagean object is moving from left to right in the real world whilethe eye is looking straight ahead. The top right image showsthe same situation but the eye is now tracking the object. Thebottom left is the same as the top left but now it is a virtualobject on a display with a limited frame rate. The bottomright image has the eye tracking the object again but this timeon a display, and we can clearly see the sawtooth causing thedescribed effect. Depending on the type of display this willpresent as judder, blur, deformations and/or color fringing.The most obvious solution is again to raise the frame rate, butit is currently not feasible to reach high enough frame ratesto eliminate these problems. Another solution is to use a lowpersistence display where instead of displaying the image forthe whole duration of a frame it is only shortly flashed, and itis left up to the human visual system to do correct interpola-tion. In turn, this can cause problems when the human visualsystem expects blur, for instance when seeing objects movingat high speeds or during saccades (fast movements of the eyewhen changing focus to a different subject).[8][14]

Figure 7. Moving objects relative to the eye

Limited resolution: The pixel density of displays has goneup a lot since the last wave of interest in VR, but individ-ual pixels can still be resolved on HMDs because they are soclose to our eyes. Anti-aliasing, which is also used on normaldisplays helps a lot but for truly immersive VR the resolutionstill needs to go up. The resolution issue does however seem

to be one people are willing to put up with (for now), unlikeother problems that actively disrupt the VR experience suchas motion sickness or headaches.[12]

Visual Displacement: VR can also suffer from this problemwe discussed in the section on AR, but this should be easilysolvable in the rendering software. It does require the devel-oper to be aware of this though.

EVALUATING HEAD MOUNTED DISPLAYSAugmented RealityThe goal of using head mounted displays with augmentedreality applications is to assist the users in their everydaytasks by improving the user’s efficiency, accuracy, awarenessand focus. In [4], Livingston describes some of the tasks thatwill most likely benefit from augmented reality. Examplesinclude visual searching, navigation and tasks requiringsituational awareness. One of the main difficulties in evalu-ating the use of AR in these tasks is the fact that current ARtechnology is fairly unoptimized. Most of the time, it consistof a single prototype model unfit for use outside of labenvironments. Also, the available hardware limitations andthe problems that come with it often impede human factorstudies. Therefore, we need to make a strong distinctionbetween technological and human factors when assessing thevalue of AR applications, which is a very difficult task.

Figure 8. The warehouse testing environment. Image by J. Tumler et al.

In [6], the user strain as effect of augmented reality assistedtasks has been studied by analysing the heart rate variabilityof the test subjects. The experiment was conducted in a ware-house environment, where the test subjects were asked to re-trieve a number of items identified by RFID tags (see figure8). Each user did the experiment twice: once with a paper listof items to retrieve and replenish, and once wearing an see-through head mounted display using an AR application to listwhich item to get next (along with its location). The heart ratevariability showed no significant differences between the twomethods (except during a short familiarization phase at thebeginning of the experiment). However, the use of an HMDinduced more eye strain (which can be classified as a tech-nological problem factor). The test runs with an HMD alsoimproved the accuracy of the given tasks.

4

Figure 9. a) The view through the head mounted display. b)Overheadview showing the correct solution. Image by M. Livingston.

In [4], Livingston conducted a test designed to overcome thedifficulties that arise due to technological factors. His solu-tion is to conduct simple, single-task based tests that use awell designed part of the user interface. His exemplary testevaluates spatial awareness: the test involved seeing objectsthrough walls by projecting them on a see-through HMD (seefigure 9). This way, the test tried evaluating depth perception.The results measured the accuracy and variance of the per-ceived depth. While the depth perception behaved the samein case of real or virtual objects, the accuracy drops signif-icantly with virtual objects, probably due to a lack of otherobjects at the same distance to compare dimensions with.

Virtual RealityThe purpose of virtual reality is to give the user the experi-ence of being inside of a world that does not physically ex-ist. Providing the input to our most heavily relied on sense,vision, makes HMDs a key instrument in providing this ex-perience. This also makes it extremely hard to perfect them,since humans are experts at detecting anything that does notlook right. Some parts of the VR experience can however not(yet) be created, such as touch and acceleration. Therefore itcan not really be tested how much a user believes they are inthe virtual world.

Some aspects that are critical to this can however be tested.In [11] R. Ruddle, E. Volkova and H. Bulthoff for instance trydifferent navigation interfaces for virtual worlds, and see howthey affect travelling performance. In their tests, participantshad to follow a path through an orthogonal grid of corridors.To travel well, a user needs to have an understanding of thespace they are in, where they are located in it and how theiractions in the real world relate to the virtual world. The papergives a nice example of a VR-related problem that lies beyondjust displaying the world convincingly.

PRESENTED PAPERSTo provide this overview and discussion of problems relatedto head mounted displays, following sources proved to bevery useful:

For augmented reality, the state of the art is described by[7]. We extracted this paper’s information about the differ-ent types of AR devices, together with the main problemsthat these devices have to cope with. Furthermore, this pa-per provided a lot of references to work that tackled specificproblems of AR devices.

The work described in [1], [5] and [10] handles about dif-ferent aspects of AR-specific problems. While [7] providesan overview of AR problems, these papers study the cause ofthese problems and provide possible solutions.

Evaluation considerations and experiments using augmentedreality devices were performed by [4] and [6].

Micheal Abrash’s blog ([12], [13] and [14]) provides a wealthof information on the topic of VR. [8] and [9] provide a moreacademic view on the problems described and also show usthat some of the problems related to VR have been known fora long time already.

If the problems described in these sources can be described as”technical”, then [11] gives us a look at one of the ”human”problems in VR. It is a good reminder that even if a HMDwere capable of fooling the eye perfectly, there would still beusability problems to be solved.

CONCLUSIONDespite the fact that head mounted displays have a long his-tory of research, quite a lot of problems still exist. In thispaper we gave an overview of these problems, both for aug-mented reality devices as virtual reality devices. Each of theseproblems was documented by describing which elements ofthe device cause them and by providing possible solutionsthat recently have been developed. When evaluating headmounted displays, it is important to make a distinction be-tween issues that arise due to the problems described (whichare mostly hardware problems) and the issues that are dueto the application itself: As [4] describes, the technologicalproblems currently still impede our abilities to assess the via-bility of most hmd applications. However, tests by [4] and[6] have shown that head mounted displays are a promis-ing technology for enhancing the user’s spatial awareness,efficiency and accuracy when carrying out everyday tasks.The recent rise in research activity, induced by projects likeGoogle Glass [2], will hopefully result in the technologicaladvancements needed to eliminate these described problems.

REFERENCES1. S. Gilson, A. Fitzgibbon, A. Glennerster, Spatial

calibration of an optical see-through head mounteddisplay, J Neurosci Methods. 173(1): 140146, 2008.

2. Google, Project Glass,http://www.google.com/glass/start/

3. Oculus VR, Oculus Rift, http://www.oculusvr.com/

4. M. Livingston, Evaluating human factors in augmentedreality systems. Computer Graphics and Applications,IEEE, 25(6):6-9, 2005.

5. T. Kollenberg, A. Neumann, D. Schneider, T.-K. Tews,T. Hermann, H. Ritter, A. Dierker, and H. Koesling,Visual search in the (un)real world: how head-mounteddisplays affect eye movements, head movements andtarget detection. In Proceedings of the Symposium onEye-Tracking Research, pages 121124, 2010.

6. J. Tumler, R. Mecke, M. Schenk, A. Huckauf, F. Doil,G. Paul, E. Pfister, I. Bockelmann, A. Roggentin, Mobile

5

Augmented Reality in Industrial Applications:Approaches for Solution of User-Related Issues. IEEEInternational Symposium on Augmented Reality, 2008.

7. D. Jackel, Head-mounted Displays. Proceedings RTMI,2013.

8. Eli Peli, Visual and Optometric Issues withHead-Mounted Displays, 1996, IS&T/OSA Optics &Imaging in the Information Age, 364, 6 pages

9. Takashi Shibata, Head mounted display, Displays,Volume 23, Issues 12, April 2002, Pages 57-64, ISSN0141-9382,http://dx.doi.org/10.1016/S0141-9382(02)00010-0.(http://www.sciencedirect.com/science/article/pii/S0141938202000100)

10. Joong Ho Lee, Sei-young Kim, Hae Cheol Yoon, BoKyung Huh, and Ji-Hyung Park. 2013. A preliminaryinvestigation of human adaptations for various virtualeyes in video see-through HMDS. In Proceedings of theSIGCHI Conference on Human Factors in ComputingSystems (CHI ’13). ACM, New York, NY, USA,309-312. DOI=10.1145/2470654.2470698http://doi.acm.org/10.1145/2470654.2470698

11. Roy A. Ruddle, Ekaterina Volkova, and Heinrich H.Blthoff. 2013. Learning to walk in virtual reality. ACMTrans. Appl. Percept. 10, 2, Article 11 (June 2013), 17pages. DOI=http://dx.doi.org/10.1145/2465780.2465785http://doi.acm.org/http://dx.doi.org/10.1145/2465780.2465785

12. Michael Abrash, When it comes to resolution it’s allrelative. November 27th 2012, Blogpost, last retrievedDecember 3rd 2013 fromhttp://blogs.valvesoftware.com/abrash/when-it-comes-to-resolution-its-all-relative/

13. Michael Abrash, Latency the sine qua non of AR andVR. December 29th 2012, Blogpost, last retrievedDecember 3rd 2013 fromhttp://blogs.valvesoftware.com/abrash/latency-the-sine-qua-non-of-ar-and-vr/

14. Michael Abrash, Why virtual isnt real to your brain.May 15th 2013, Blogpost, last retrieved December 3rd2013 from http://blogs.valvesoftware.com/abrash/why-virtual-isnt-real-to-your-brain/

6