Everyday Virtual RealityJames Walker

Michigan Technological University

02/27/15

Table of Contents1 Introduction.............................................................................................................................................22 Thesis......................................................................................................................................................33 Background.............................................................................................................................................3

3.1 Head-Mounted Displays..................................................................................................................33.2 Usage Issues....................................................................................................................................43.3 Window Managers...........................................................................................................................43.4 Reading Documents and Navigating Data......................................................................................7

4 Completed Work.....................................................................................................................................84.1 The Effects of Minification in HMDs.............................................................................................84.2 Flow Field Visualization with FlowTour.........................................................................................94.3 The Immersive Visualization Studio.............................................................................................10

5 Action Plan............................................................................................................................................105.1 Combining Real and Virtual Information......................................................................................11

5.1.1 Development..........................................................................................................................125.1.2 User Study.............................................................................................................................12

5.2 Task Switching..............................................................................................................................135.2.1 Development..........................................................................................................................135.2.2 User Study.............................................................................................................................14

5.3 Reading Documents......................................................................................................................155.3.1 Development..........................................................................................................................155.3.2 User Study.............................................................................................................................16

6 Summary of Experiments......................................................................................................................177 Timeline to Completion........................................................................................................................188 References.............................................................................................................................................19

1 IntroductionFor the first time, virtual reality interface technology is achieving sufficiently low cost and ease of availability for widespread home and office use, as illustrated most prominently by the Oculus Rift. However, to date there has been relatively little investigation into the “mundane” applications of virtual reality, such as its possible uses on a home or office desktop environment. This creates the possibility of new technology flooding the market without any clear uses for it outside of the obvious entertainment applications.

These potential uses are worth investigating because virtual reality interface technology could provide new and original approaches to the most fundamental tasks users perform on their computers, such as application management and reading documents. The goal of this work is to develop software optimized for V.R. interface technology (specifically, head-mounted displays) for use with everyday home and office tasks and to perform experiments to compare the efficacy of these techniques with traditional interaction methods. This work will serve as a preliminary investigation into the best way to incorporate virtual reality into users' everyday desktop (or laptop) experiences.

This is not a merely theoretical area of investigation. Users have already expressed interest in utilizing head-mounted display (HMD) technology to manage their desktops and applications. Here are some user comments in response to an experimental HMD window manager [34] posted on Reddit [47]:

• “Wow. This is something I'd hoped to see the Rift used for. I'm looking forward to replacing my desk full of monitors with a single headset.”

• “Imagine having an office workspace with no physical displays at all, just a Rift and you're done… you're surrounded by a relaxing environment like a some waterfalls in a jungle amidst which your terminal windows float around, awaiting your input”

• “imagine that...instead of paying a grip for 3 30" monitors or whatever you could sit there and design the ultimate virtual workstation or entertainment center... so cool.”

Other examples of burgeoning HMD usage can be found. For instance, Bloomberg has created a prototype virtual terminal for use with the Oculus Rift [38]. Or consider the fact that most initial purchases of consumer HMDs for home use will be for entertainment purposes; i.e. virtual reality video games. Switching between using the HMD for games and a standard monitor for other desktop applications could be burdensome. An HMD-optimized window manager would allow users to switch between their entertainment and utility applications without needing to also switch display hardware.

The products of this work will be improved knowledge of how to design user interfaces for use with HMDs in a home or office environment and the prototype software created in the process of conducting these investigations. The software is not intended to be release quality by completion of the dissertation, but to serve as a test platform and functional proof of concept. A series of user studies will evaluate the effectiveness of the software and locate areas where the design needs improvement.

2

2 ThesisVirtual reality interface technologies can be utilized to enhance users' computer experiences by enabling routine tasks to be carried out in ways that are more efficient or enjoyable than traditional interfaces

3 BackgroundThis work will need to address physical obstacles that arise while using an HMD for home and office computing tasks as well as methods to harness HMDs' capabilities for routine computing tasks. I have chosen to focus on window management, document reading, and data navigation. A brief survey of the background and most relevant related work in these areas follows.

3.1 Head-Mounted Displays

The virtual reality technology most fundamental to the proposed research is the Head-Mounted Display (HMD). An HMD is a helmet or visor worn by the user with two screens, one for each eye, so that a stereoscopic “true 3D” image may be displayed to the user. This is achieved by displaying the same image in each screen, but offset by a distance equal to the distance between the user's eyes, mimicking how human vision perceives the world.

HMDs can be opaque or see-through. In a see-through HMD, the screens are transparent so that the user can see the real world as well as what is being displayed on the screens. However, see-through HMDs often suffer from brightness problems that make them difficult to use in variable lighting conditions. Most opaque HMD designs block out the real world so that the user can only see the screens, thereby providing an immersive experience.

HMDs are usually used in conjunction with tracking systems. By tracking the user's position or orientation (or both), the system can allow the user to move naturally via locomotion and by turning their head and body, and update the graphical display accordingly. This allows for natural exploration of virtual environments without needing to rely on a keyboard, mouse, joystick, and similar interface hardware. Positional tracking is often accomplished by attaching markers (such as infrared markers) to the HMD or the user's body and using multiple special cameras to track the location of these markers in 3D space. Orientation tracking can be accomplished using an inertial tracker, which uses a sensor to detect velocities on three axes. Some systems use a combination of optical and inertial tracking, and other tracking techniques (e.g., magnetic) also exist. The output from the tracking systems is fed into the computer rendering the graphical display so that it can update the scene. Filtering is usually necessary to make the data usable since it comes in the form of noisy analog measurements.

The Immersive Visualization Studio at Michigan Tech includes a high-end Vicon tracking system which uses an array of infrared cameras and markers to track both position and orientation of multiple objects with sub-millimeter precision at an update rate of over 100 Hz.

I have chosen HMDs as the technology to study because they have become inexpensive enough for consumer use and they are small enough to easily be used in a home or office environment. Other virtual reality technologies, such as CAVE and display walls, remain prohibitively expensive for the average user and take up too much space to be used in most enclosed environments.

3

3.2 Usage Issues

Various obstacles must be overcome in order to enable users to use HMDs as the primary display hardware while interacting with their computers. For example, while wearing an HMD, users will be unable to see the real world, unless they are using a see-through HMD. However, as discussed, see-through HMDs often suffer from brightness problems that make them ill-suited for use in variable lighting conditions. Addressing this issue might involve some form of augmented reality; that is, the blending of the real and virtual worlds. It might be useful to track the real-world coordinates of certain physical objects (such as a keyboard) and display graphical representations of them in the virtual world. This alignment of real and virtual objects is known as registration, and is a well-studied problem [2, 3]. It would be valuable to compare such an approach to see-through HMDs and see which yields better user performance.

Another issue is that users cannot see their keyboard while wearing an HMD. For users who can touch-type, this will not be an obstacle as long as they can find the home keys. However, it is not necessarily a safe assumption that the vast majority of users can touch-type. There are some suggestions that touch-typing is a declining skill because of the ubiquity of touch screen keyboards on mobile devices [43]. If users do not have time to learn how to touch-type, and are not using a see-through HMD, then they will need some kind of special aid to be able to type while wearing an HMD.

3.3 Window Managers

The task of managing multiple applications on a traditional desktop is accomplished using a special type of application called a window manager. Window managers come in multiple styles. The most common types of traditional window managers are stacking and tiling. Stacking window managers allow users to place and size windows at will, including placing windows on top of each other so that their areas overlap. By contrast, tiling window managers do not allow overlap and aid the user in organizing windows into tiles on the screen. Both of these styles are fundamentally 2D in nature, designed for use with flat screens, and do not leverage the true-3D stereoscopic display and theoretically unlimited display area provided by HMDs.

There are also compositing window managers, also called compositors. Compositors store applications' visual contents in a graphical buffer so that they can be subjected to sophisticated graphical manipulations (e.g., using OpenGL) in order to achieve striking visual effects and enable three-dimensional interface elements. Rather than a category in itself, compositing is more of a property of window managers. A window manager could be both stacking and compositing, for example.

Figure 1. Examples of stacking, tiling, and compositing window managers.

4

On current Linux systems, window managers must utilize the X protocol and are typically coded either against the X library itself or an intermediate API such as XCB. Because of the X library's age, it was not designed with 3D graphical capabilities in mind. This capability has been added in an ad-hoc manner using various extensions, which makes writing compositing window managers for X a convoluted process. Alternatives exist, such as the Wayland protocol, but these are in experimental stages and thus do not yet serve as a stable development platform [46].

Users are always striving for more efficient ways to simultaneously manage large numbers of applications. Many users prefer large displays, or multiple displays, for this purpose. Research has shown that users are often more productive when given larger display areas to work with, yet there are also disadvantages compared to a traditional monitor; for example, large amounts of time spent repositioning and resizing windows, accidentally closing windows, and difficulty navigating a large display area with a mouse and keyboard [6, 7, 21, 35, 39]. These findings simultaneously demonstrate the utility of large display areas and the problems that arise from using traditional interface technologies to present large displays to the user.

HMDs, with their ability to display a virtual environment that entirely surrounds the user, present an opportunity to move beyond the limitations of flat monitors. Some companies and researchers are already recognizing the potential in this area. For example, Bloomberg has created a prototype virtual terminal for use with the Oculus Rift [38]. Reiling developed a window manager called Motorcar for use with HMDs [34]. Because Motorcar bears the closest resemblance to one of the intended products of this research, it warrants a more detailed examination.

Motorcar is fundamentally a compositing window manager developed using the Wayland protocol. The primary objectives of Motorcar are to enable the free arrangement of application windows in three-dimensional space, to provide transparent support for both 2D and 3D applications so that applications displayed in Motorcar can be developed in a hardware-agnostic manner, and to be designed for use with HMDs as the primary display device, rather than flat monitors. However, while Motorcar serves as an effective tech demo and proof-of-concept, its lack of restrictions on placement and positioning of windows is likely a drawback rather than an asset. Some of the user comments in response to a video of Motorcar in use echo this sentiment, with such comments as, “[T]hat looks excruciating,” and “Neat to look at but impossible to use for any real tasks” [47]. While Motorcar is a step in the right direction, an HMD-optimized windowing system needs a greater focus on usability and user studies to evaluate the effectiveness of different interface approaches.

Another similar project is the Personal Cockpit developed by Ens et al [15, 16]. The Personal Cockpit is a virtual window manager designed for use with see-through HMDs; and unlike the vast majority of window manager research explored, this work includes a user study evaluating the system's effectiveness, with favorable results. However, it differs from my proposed work in that it relies on the user being able to manipulate the virtual windows by direct touch, which necessitates a system to track the user's hands, placing the system out of reach for many consumers, who do not have ready access to such tracking systems.

The Virtual Pad by Andujar and Argelaguet [1] is a system with user-configurable virtual tablets (called virtual pads) which allow a disparate mapping between the user's movements on the pad area and the resulting input to 2D windows being displayed in a 3D space. They performed a user study which found that there was no significant difference in speed or accuracy between more traditional direct

5

input methods and the virtual pads, while the pads allow for greater comfort to the user (presumably, although the study did not measure user comfort). This differs from my work in that my proposed systems allow for using traditional input methods (i.e., keyboard, mouse) so that hand tracking is not required.

Other window management systems with some similar features, but different goals, have been developed. Robertson et al [36] developed a 3D window manager for Microsoft Windows called the Task Gallery that utilizes users' spatial memory and bears some similarities to a virtual environment desktop. Topol [42] presents a similar system for the X11 windowing system that organizes application windows in a virtual environment-like space. A similar but more sophisticated system called 3DWM (Three-Dimensional Workspace Manager) was developed by Elmqvist [14]. However, none of these systems are appropriate for desktop HMD use. Both the Task Gallery and Topol's system were designed for use with traditional monitors and do not support sensor input as a control mechanism. 3DWM is essentially a hardware abstraction layer and toolkit to facilitate development and research of 3D user interfaces rather than an effective interface in itself.

Figure 2. Screen capture from a video of the Motorcar window manager in use.

There has also been similar work in the field of augmented reality. Feiner et al [17] developed an augmented reality window manager for use with see-through HMDs, but their intent was to create a virtual heads-up system rather than an effective desktop window manager. DiVerdi [10, 11] developed an augmented reality window manager called ARWin with a focus on displaying applications as though they were an extension of the user's physical desk.

Finally, another subgenre of VR research that bears some similarity to my proposed work is virtual/augmented reality menu systems. Although menu systems in isolation are not as robust as a full window manager, most window managers include some sort of menu system as part of their functionality. Common types of menu systems include floating menus, which are similar to standard 2D menus familiar to most users except that they float in 3D space; pen and tablet menus where interface components are placed on a tablet or similar surface and interacted with via a stylus or similar device; and gesture-based menus which respond to user movements, often aided by a tracked device such as a glove [4]. Gesture-based menus have received a particularly large share of attention in recent years. A few examples include the Pinch Glove by Bowman and Wingrave [4]; the WeARHand system, an AR gesture-based selection system that tracks users' bare hands with HMD-mounted RGBD cameras [19]; and the vDesign system for use with CAVEs [32]. More exotic VR menu systems have also been

6

implemented and tested, such as pie menus activated by a 6DOF pointing device [18]. Some of these techniques might be used as inspiration when designing parts of the user interface for a VR window manager.

Overall, while HMDs show obvious potential for desktop use, research into this area it still preliminary in nature, with few studies objectively evaluating effective techniques. This is an area ripe for investigation.

3.4 Reading Documents and Navigating Data

Possibly excepting devices built specifically for reading (i.e., eReaders), reading on a computer screen is widely regarded as an inferior experience to reading on paper. For example, many users find using a mouse and keyboard to navigate a document via scrolling to be cumbersome and “annoying” [33]. When reading documents from physical books, readers are better able to remember the location of specific content by utilizing spatial memory; that is, by remembering the content's physical location on a page and approximate location within the document as a whole [20, 33]. However, scrolling has been found to weaken spatial memory [27]. Lovelace and Southall [28] had college students read passages in continuous form from a scroll, which is similar to how documents are usually read digitally, and found that this significantly lowered users' word recall. In sum, many studies have demonstrated that there are significant disadvantages when reading documents through traditional computer interfaces, or even physical media that work similarly to computer interfaces.

More generally, document viewing can be considered a special case of digital data management and navigation. Card, Robertson, and Mackinlay introduced the concept of information workspaces and described the implementation of an experimental system that used 3D rooms to present information to the user [5]. Williams et al showed that “spatial learning is similar in both real and virtual environments” [48]. Yost, Haciahmetoglu, and North showed that users can work with large data sets more efficiently with lots of display space [50]. DeLine et al explored using spatial memory for navigating source code while programming and found that it helped programmers understand and navigate code more easily [9]. This capability is now used in some popular text editors, such as Sublime [40].

Figure 3. Screen capture of the Sublime text editor. Note the use of minimaps (code thumbnails).

7

Based on these findings, HMDs, which support natural movement, stereoscopic vision, and presenting virtual environments of nearly unlimited size, appear to provide an opportunity to enhance users' experiences navigating documents and other data sets on their home or work computers.

4 Completed WorkTo date, I have conducted published research exploring improvements in user interfaces in general, with an emphasis on using virtual reality technologies. Although some of these experiments are not directly related to desktop HMD applications, they do share the overall goal of making HMDs and 3D desktop displays more useful. My proposed dissertation research follows this general line of study, with a narrower and deeper focus. An overview of my previously published work follows.

4.1 The Effects of Minification in HMDs

I have assisted and led several studies researching the effects of minification in HMDs. It is a widely known problem that users tend to underestimate distances in HMDs as compared to the real world [25, 26, 30, 37, 49]. Minification is a technique of increasing the user's field of view in software without increasing the size of the physical display, with the effect of making everything look smaller. Minification has been found to improve users' distance judgments in virtual environments [24, 25], but due to the nature of the technique, it could have undesirable side effects. I contributed to several studies to explore some of those potential side effects.

The first study explored the effects of minification on verbal- and action-based distance judgments in HMDs [51]. The study consisted of a series of trials in which participants were shown targets on the floor in a virtual environment and indicated perceived distance to the targets with either verbal reports or blind walking. There were two conditions, normal vision and minified vision. The result of the study was that minification caused users to judge distances more accurately as measured by blind walking, but did not have an effect on users' verbal reports of the distance (they were still underestimated). This was a positive outcome since it suggested that minification helps correct users' distance underestimation without changing their conscious perception of spatial relationships. I helped run the experiments and assisted with editing the paper for submission.

Figure 4. Virtual world in normal (left) and minified (right) conditions from experiments in [51].

8

I then led a follow-up study exploring the effects of minification on gap affordances; that is, users' perception of the widths of openings and whether they can pass through those openings [45]. I designed the study, developed the software, ran the experiments, and wrote up the results. In this study, 12 participants were repeatedly shown two poles at varying distances and reported whether they believed they could fit between the poles without turning. There were two conditions, normal and minified view. The study used a within-subject design, with half of the participants viewing normal view first and half minified first. No statistically significant difference was found between the conditions. This provided further evidence that minification does not affect users' conscious perception of spatial relationships, which is a desirable characteristic of perceptual manipulations intended to improve user performance.

4.2 Flow Field Visualization with FlowTour

Next, I contributed to several studies to evaluate the effectiveness of FlowTour, software designed to provide an innovative perspective of flow field visualizations with automated internal tours [29, 44]. Although this research represented a temporary departure from studying virtual reality technologies, it held the common thread of designing experimental 3D user interfaces and evaluating their effectiveness with formal user studies. I led the design of the studies, ran the experiments, analyzed the data, and helped write the paper.

In these studies, users were presented with views of six flow field data sets in three conditions (FlowTour, random internal path, guided external path), then asked to identify critical features in the data sets and to answer questions about them. The study had 31 participants and used a between-subject design. The result was that FlowTour helped users identify critical regions better than the other conditions, but did not have a statistically significant effect on user performance when answering general questions about the data sets. Our conclusion was that the software's automated tours, which were specifically designed to show critical regions (especially internal critical regions which can be difficult to see with an external view), succeeded in this goal; but that the internal tours did not improve users' understanding of flow field data sets in general.

Figure 5. An image from the FlowTour software [29].

9

4.3 The Immersive Visualization Studio

Concurrently and after the work with FlowTour, I resumed my research focus on virtual reality technologies by working with the display wall and Vicon tracking system in the Immersive Visualization Studio (IVS). I first developed a distributed graphics rendering framework which has been used in other research groups' projects and has also been subsequently improved upon to the point where it now constitutes a user-friendly API that makes the networking issues required for efficiently rendering graphics on the display wall nearly transparent.

Figure 6. Screen capture of software developed for [23].

Following this work, I assisted in developing interactive sonification software, the Immersive Interactive Sonification Platform (iISoP), which utilizes the display wall and the Vicon tracking system in the IVS lab to enable dynamically reactive sound in response to user movements [23]. I also used this software as a basis to create another application which used the Vicon tracker to enable drawing in three-dimensional space, work that was used in another paper [22]. These projects did not include user studies or report experimental results; rather, the published papers served as descriptions of innovative interface systems and their potential applications.

My work in the IVS lab is the most directly applicable to my proposed dissertation research, as the software I've developed for it could be used as the basis for the Combining Real and Virtual Information study described in section 5.1.

5 Action PlanThe core of the work will be centered on a custom-designed windowing system optimized for use with virtual reality hardware which I will develop, the Fish Out of Water or FOW system. This system will serve as the basis for investigating the efficacy of virtual reality in a home or office environment.

The investigation will be divided into three main phases. In the first phase, potential obstacles in home and office V.R. use will be researched, along with the effectiveness of various countermeasures to those obstacles. Once the best approach has been determined, it will be used to mitigate potential interface problems in all remaining research.

10

In the second phase, the effectiveness of the FOW system for task switching will be studied by comparing it to more traditional windowing systems. Managing and switching between applications is one of the most fundamental computing tasks carried out by users, so it would be highly beneficial if virtual reality interface technology could be used to improve this task.

Finally, a variant of the FOW system will be used for document reading, which is another very common task carried out by users. Traditional methods of document navigation (i.e., using scrollbars) can be cumbersome, particularly when reading long or complex documents. Using an HMD may present an opportunity to improve document navigation by “wallpapering” a virtual surface with pages from the document, thus allowing the user to utilize spatial memory and navigate to any part of the document merely by rotating his head orientation.

5.1 Combining Real and Virtual Information

Even assuming that virtual reality technology improves users' experiences performing some routine computing tasks, certain obstacles are sure to arise. For example, if the user is not using a see-through HMD, they could lose track of their orientation in relation to their physical environment. This would be a problem if the user loses track of their input devices (keyboard, mouse, etc.) and needs to waste time finding them again. Or, imagine the scenario where a user knocks over a beverage which spills into their keyboard or on important documents. Clearly, it is not desirable for the user to lose all cognizance of their physical environment while performing routine computing tasks.

Additionally, users who are unable to touch-type would have difficulty using their keyboard while wearing the HMD. See-through HMDs might alleviate these difficulties, but investigation is needed to determine whether their brightness problems interfere with users' ability to use the system. These issues must be addressed before focusing on less critical problems like the best kind of user interface, so I will study this aspect before moving on to subsequent development phases.

Various solutions are possible. A solution which does not rely on a see-through HMD or external video feed would have to rely on augmented reality. For example, the physical location of the keyboard and mouse could be tracked and displayed as ghosted or wireframe objects within the virtual world. Or, since users might not have access to tracking hardware sufficient for this purpose in their homes and offices, positions of relevant objects could be pre-calibrated instead of live-tracked. Either way, this tracking can be easily accomplished using proven registration techniques [2, 3]. In this scenario, the problem of non-touch-typing users trying to type on their keyboards is potentially more challenging, but might be mitigated by displaying a virtual keyboard on the screen and having the keys light up as the user presses them, while still using the real keyboard as a haptic interface.

Alternative solutions become possible if the user has a camera mounted on their HMD which can provide a video feed of the real world. In this case, a live camera feed could be shown in the corner of the display, or a special command (such as the user tilting his head) could toggle between showing the virtual world, the feed from the camera, or both at once. With a see-through HMD, a command could be used to turn the graphics on or off to provide a better view of the real world when necessary.

Another potential issue with using an HMD for home and office computing is health effects, because habitual, long-term HMD use has not been well-studied. Only one study has been found evaluating the effects of HMD use over a 24-hour period [41]. The study consisted of a self-experimentation report in

11

which one of the authors remained immersed in a virtual environment almost continuously for 24 hours. The subject reported some uncomfortable simulator sickness, but no seriously harmful effects. To my knowledge, no other studies about the health effects of long-term HMD use have been conducted. Despite this area being an open research question, it was decided to leave this issue aside for the purposes of this work; partially due to the extreme difficulty of designing an appropriate study for this purpose, and partially because, even without a study, any health issues will rapidly become apparent with the widespread adoption of common HMD use.

5.1.1 Development

Using a see-through HMD or camera feed, development would be a simple matter of programming a command to toggle between different display modes. For the augmented reality solution, tracking physical objects could be accomplished by utilizing the Vicon system already installed in the Immersive Visualization Studio. I have already done extensive work with the system in previous research [22, 23], so utilizing that experience would make designing new software a simple proposition. The software will display a virtual world with the locations of physical objects overlaid on it utilizing registration so that users can see the virtual world and the positions of tracked objects simultaneously.

If it seems desirable to address the problem of non-touch-typing users in the augmented reality solution, I will develop software that renders a virtual keyboard to the screen and has individual keys flash when the user presses them to aid the user in finding and pressing the right keys despite not being able to see the keyboard.

5.1.2 User Study

I will conduct a user study to measure the detrimental effects of HMD use and the effectiveness of the proposed solutions. The study will have a control condition where the user receives no feedback about the physical world, and two or three experimental conditions. These experimental conditions will include registration software developed for use in the IVS lab and a see-through HMD. A follow-up study using an HMD-mounted camera with a live video feed, with the ability to toggle between display modes, may be used in a follow-up study if it seems worth pursuing after the first study.

For this study, the user is presented with a simple virtual environment and is asked to locate targets within the virtual environment by moving their head. In between locating virtual targets, the user is periodically asked to indicate where they believe certain physical objects are located in the real world, without removing the HMD. This will be accomplished by having the user wear a tracking glove and point at the tracked object. We will use the tracking system to measure the accuracy of their pointing. Alternatively, the user might be asked to physically handle the tracked object (keyboard, mouse, etc.) in some way to more accurately mimic a realistic use scenario. In this task, the metric used to evaluate users' performance will be their speed and accuracy in locating physical objects.

A potential follow-up study would address the typing problem using the same conditions as the first study. For this study, the user will be presented with a brief document and asked to type it without removing the HMD. The metric used to evaluate users' performance will be how accurately they transcribe the document (i.e., number of typos) and how long it takes them to transcribe the document

12

(time elapsed). In this case, a pre-test would be necessary to control for variable typing speeds, having the user type another document to measure their baseline typing speed.

The hypothesis for both tasks is that users will be inept in the control condition and significantly better in the experimental conditions.

5.2 Task Switching

A fundamental task that users routinely perform on their computers is switching between different applications. In the context of an operating system, a window manager is often employed to facilitate efficient task switching. This presents an opportunity to find ways to utilize the unique properties of HMDs to examine original window management techniques. As discussed in section 3.3, no existing window managers meet the requirements of providing a fully 3D interface, being designed for use with an HMD, and having user studies to evaluate their usability.

To this end, I propose the development of a virtual reality-optimized window manager called Fish Out of Water (FOW). FOW will present the user with a virtual room where open but inactive applications (the equivalent of “minimized” applications in other window managers) appear on the walls of the virtual room as thumbnails. It has been shown that landmarks benefit navigation performance in virtual reality [8], so the virtual room will contain landmarks to aid the user's ability to remember where various applications are located. The user will view the room through an HMD and be able to view any part of the room simply by turning his head. Clicking on a thumbnail will bring the corresponding application into focus, in which case it will fill most of the display, utilizing stereoscopic vision to appear as if it is floating in true 3D space.

5.2.1 Development

Significant effort has already been expended in the development of the FOW system. Both the X protocol and Wayland were investigated as possible frameworks on which to build a virtual reality window manager. However, X's age makes 3D graphical rendering extremely convoluted and Wayland is still in an experimental stage. Development using either platform is therefore a very time-consuming process. Because the objective of this research is not to create a release-quality product that can be deployed “out of the box” on existing systems, but rather to explore and evaluate the effectiveness of HMD interface techniques for common computing tasks, the time-intensive nature of X and Wayland development resulted in both being rejected as development platforms for FOW.

Instead, a prototype FOW system was developed in Java using an OpenGL back-end for graphical display and AutoHotkey scripts to interact with the operating system and control other applications. This approach cut development time to a small fraction of what would be required with X or Wayland, with the drawbacks that FOW can only be deployed on Microsoft Windows systems and is quite demanding on hardware resources. The prototype utilizes stereoscopy to achieve true 3D using an HMD and also applies lens distortion correction with a GLSL fragment shader. The prototype also interfaces with the Oculus Rift using a relay program developed with the Oculus SDK2 that forwards the HMD's orientation data to the window manager via UDP packets. The basic features of application manipulation and thumbnailing are already in place. However, many minor details and an attractive graphical representation have yet to be developed; these will be added to the prototype during this stage of the research.

13

5.2.2 User Study

I will conduct a user study to test FOW's suitability for task switching. While there exists a rich body of research devoted to cognitive task switching in general (see, e.g., [31] for an approachable discussion of the topic), we are concerned with the much narrower question of task switching in the context of windowing systems; i.e., which system enables users to be the fastest and most accurate when managing multiple applications, and which system feels most convenient and pleasant to use?

Unfortunately, very little of the research dedicated to window managers seems to include objective evaluations of the their effectiveness through user studies, focusing instead on novel system features. As one exception, Robertson et al [36] performed several small user studies to evaluate the Task Gallery. Their studies evaluated users' speed and accuracy when performing various tasks with the Task Gallery; but they did not compare these results to other window managers, as the tasks evaluated pertained to features exclusive to the Task Gallery. Their studies also included subjective user satisfaction questionnaires. Another exception is the Personal Cockpit [15].

Hutchings et al [21] conducted a study about behavioral differences between single and multiple monitor users, which has some relevance since an HMD enables users to utilize a conceptually greater amount of display space for window management. They conducted their study by installing automated logging software on participants' computers and tracking participants' window management behavior over a period of three weeks. Unfortunately, such an approach is impractical for evaluating FOW since most users still do not possess an HMD for use with their home or office computers.

The FOW user study will take an approach somewhat similar to the Task Gallery study, but will also compare FOW's performance to traditional window managers. The study will have a control condition using a stacking window manager, and multiple experimental conditions using FOW with different settings. While the different interface settings to be evaluated have not yet been finalized, several possibilities are being considered:

• How “sticky” is the window currently in focus? Does it always remain in the center of the user's vision, does it float freely in 3D space (as in Motorcar), or somewhere in between?

• How are inactive windows laid out/sorted/organized?• When a new window appears, what happens to the old windows?• How does the system perform with various input techniques; e.g., mouse and keyboard vs.

gesture-based?

The study will consist of one task. The experiment is designed to simulate a realistic task that involves switching between several applications many times, similar to the user study carried out to evaluate the Personal Cockpit [15]. In this task, the user is required to edit a document according to instructions from a fictional superior. The applications required to carry out the task include the following:

• The document being edited.• A web browser pointing at an email from the user's fictional superior, which contains

instructions for editing the document.• An image editing program. Part of the task requires editing an image and then pasting it into the

document.

14

• A window describing computer settings. The user needs to reference this data for placement in the document.

• A performance report. The user needs to reference this data for placement in the document.

In addition to the applications required for completing the task, there are also several “dummy” windows which have no bearing on the task and effectively serve as noise (e.g., a music player, a calculator, etc.).

User performance will be evaluated based on how quickly and accurately the user completes the required task. It is hypothesized that users might perform better with the stacking window manager due to greater familiarity. However, the hope is for FOW to perform reasonably well, and to try several alternative approaches to the virtual interface to see which one performs the best. The study will also include a subjective user satisfaction questionnaire.

5.3 Reading Documents

Based on the findings discussed in section 3.4, I propose that virtual reality interface technology can be used to improve users' experiences when reading documents. This may be particularly evident when parsing a document in a non-linear manner; e.g. when flipping back and forth between reading the text and checking an index or bibliography. In these situations, using a scrollbar or flipping through digital pages is an inefficient interface for document navigation, and users can easily lose their place.

Given the success of minimaps in aiding programmer comprehension of program code [9] and this feature's adoption in some popular text editors such as Sublime [40], a similar feature might help with document navigation in general. Being able to see the entire document at once in a miniaturized fashion could improve users' speed and ability to recall the locations of important areas (such as where they were last reading) when parsing documents non-linearly. For large documents, a greater display area than is offered by standard monitors, even those with high resolution, might be required in order to make effective use of this technique. In this case, a display wall-like interface could be helpful for showing the entire document without excessive miniaturization. Since most users do not have access to a display wall at home or in their workplace, this is where HMDs come into play.

5.3.1 Development

I will use FOW as a baseline to develop an application that uses an HMD for document navigation. This variant of FOW will be called VDV, the Virtual Document Viewer. Whereas FOW places thumbnails of applications on the walls of the virtual room and displays the application window when a thumbnail is clicked on, VDV will place thumbnails of document pages on the walls, and will display a page in large print when a thumbnail is clicked on.

The basic display code will be the same between both programs. Because VDV does not need to interact with the operating system, it will dispose of the AHK scripts used by FOW. Instead, it will have a mechanism for converting parts of a document into textures so that they can be used seamlessly as part of the 3D display.

15

5.3.2 User Study

Once VDV is ready for use, I will conduct a user study to test the efficacy of the techniques discussed above. The study will have two or three conditions: one control condition using the traditional document navigation techniques of scrollbars, one experimental condition using VDV; and possibly a second experimental condition using VDV on a large monitor (to see if the HMD is needed).

Because this study involves reading documents, differences between subjects' reading speeds will have to be accounted for. Subjects will be asked to read a 1000-word document at a natural pace, and the time taken will be measured. Subjects will then be asked a few questions to confirm their comprehension of the material read, drawing on work by Dyson and Haselgrove [12, 13] to ensure that the questions measure the participants' comprehension effectively.

The study will consist of two tasks. In the first task, users will be presented with a large document and will be asked to locate specific chapters or sections within the document. Sometimes they will be asked to locate the same section more than once, to see whether spatial memory enables them to recall the location of sections they have visited before more quickly with VDV. This task will bear some similarity to spatial memory trials conducted by DeLine et al [9] regarding the effectiveness of minimaps (called code thumbnails in their study) for navigating source code.

In the second task, the users will be asked to read part of a research paper and answer a questionnaire regarding the citations that were used in the paper. This task will require frequent navigating to the end of the document (to look up citations) and to the middle of the document (to resume reading where the user left off), with the idea being that this frequent back-and-forth navigation will be easier with VDV's layout. This part of the study will once again draw on Dyson and Haselgrove's work [12, 13] to ensure the validity of the comprehension questions.

The metrics used to evaluate user performance on these tasks will be how many questions they get right and how long it takes the user to answer the questions. The hypothesis is that users will be able to complete the tasks more quickly using VDV because of enhanced non-linear navigational speed.

The study will also include subjective questions regarding users' satisfaction with the navigation techniques.

16

6 Summary of Experiments

Experiment Tasks Conditions Subjects

Combining Real and Virtual Information #1: Tracking Objects

Locate targets in the real world while using a standard desktop system while wearing an HMD

Control: Opaque HMD w/o software supportExperimental #1: Opaque HMD w/ registrationExperimental #2: See-through HMDExperimental #3: Camera w/ live video feed (potential)

20-40 (between subject) or 12-24 (within subject)

Combining Real and Virtual Information #2: Keyboard Use (potential follow-up study)

Type a brief document while wearing an HMD

Same as previous Same as previous. Note: In order to alleviate the burden of recruiting research subjects, this study could potentially be merged with the previous study.

Desktop Management

Edit a document in a way that requires switching between several applications (simulating desktop management)

Control: Stacking window managerExperimental: Custom-made FOW system using various layouts and interface techniques as discussed in section 5.2.2 (one condition per variation of FOW)

20 (between subject) or 12 (within subject)

Reading Documents 1. Locate specific sections in a large document

2. Read part of a research paper and answer a questionnaire about citations used in the paper

Control: Standard document viewing programExperimental #1: Custom-made VDV system w/ HMDExperimental #2: VDV on a large monitor (potential)

20-30 (between subject) or 12-18 (within subject)

Follow-up Experiment(s) (potential)

In the event that one of the preceding experiments uncovers results that would be interesting to follow up on, another experiment could be run, possibly using a previous experiment as a baseline with some modifications.

17

7 Timeline to CompletionSpring 2015 • Give dissertation proposal

• Run Combining Real and Virtual Information experiments• Write/submit research paper corresponding to CRAVI• Work on FOW software• Work on VDV software

Summer 2015 • Complete FOW software• Complete VDV software

Fall 2015 • Run Task Switching experiments• Write/submit research paper corresponding to Task Switching

Spring 2016 • Run VDV experiments• Write/submit research paper corresponding to Reading Documents• Begin work on dissertation document

Summer 2016 • Complete dissertation document• Revise dissertation document to committee's satisfaction

Fall 2016 • Dissertation defense

18

8 References[1] Andujar, C., & Argelaguet, F. (2007). Virtual Pads: Decoupling Motor Space and Visual Space for Flexible Manipulation of 2D Windows within VEs. In 3DUI (p. 25).

[2] Azuma, R., Balliot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B. (2001). Recent Advances in Augmented Reality. Computers & Graphics, November 2001.

[3] Azuma, R., & Bishop, G. (1994, July). Improving static and dynamic registration in an optical see-through HMD. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques (pp. 197-204). ACM.

[4] Bowman, D. A., & Wingrave, C. A. (2001, March). Design and evaluation of menu systems for immersive virtual environments. In Virtual Reality, 2001. Proceedings. IEEE (pp. 149-156). IEEE.

[5] Card, S.K., Robertson, G.G., & Mackinlay, J.D. (1991). The Information Visualizer, an Information Workspace. In Proceedings of CHI '91, Human Factors in Computing Systems (New Orleans, LA, April 27-May 2, 1991), ACM, 181-188.

[6] Czerwinski, M., Robertson, G., Meyers, B., Smith, G., Robbins, D., & Tan, D. (2006, April). Large display research overview. In CHI'06 extended abstracts on Human factors in computing systems (pp. 69-74). ACM.

[7] Czerwinski, M., Smith, G., Regan, T., Meyers, B., Robertson, G., & Starkweather, G. (2003). Toward characterizing the productivity benefits of very large displays. In Proceedings of Interact 2003 (Vol. 3, pp. 9-16).

[8] Darken, R. & Silbert, J.L. (1996). Navigating large virtual spaces. International Journal of Human-Computer Interaction, 8, 49-72.

[9] DeLine, R., Czerwinski, M., Meyers, B., Venolia, G., Drucker, S., & Robertson, G. (2006, September). Code thumbnails: Using spatial memory to navigate source code. In Visual Languages and Human-Centric Computing, 2006. VL/HCC 2006 (pp. 11-18). IEEE.

[10] DiVerdi, S. (2007.) Towards Anywhere Augmentation. (Doctoral dissertation). ProQuest.

[11] DiVerdi, S., Nurmi, D., & Höllerer, T. (2003, October). ARWin-a desktop augmented reality window manager. In Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality (p. 298). IEEE Computer Society.

[12] Dyson, M., & Haselgrove, M. (2000). The effects of reading speed and reading patterns on the understanding of text read from screen. Journal of Research in Reading, 23(2), 210-223.

[13] Dyson, M. C., & Haselgrove, M. (2001). The influence of reading speed and line length on the effectiveness of reading from screen. International Journal of Human-Computer Studies, 54(4), 585-612.

19

[14] Elmqvist, N. (2003). 3Dwm: A platform for research and development of three-dimensional user interfaces. Technical Report CS: 2003-04, Chalmers Department of Computing Science.

[15] Ens, B. M., Finnegan, R., & Irani, P. P. (2014, April). The personal cockpit: a spatial interface for effective task switching on head-worn displays. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems (pp. 3171-3180). ACM.

[16] Ens, B., Hincapié-Ramos, J. D., & Irani, P. (2014, October). Ethereal planes: a design framework for 2D information space in 3D mixed reality environments. In Proceedings of the 2nd ACM symposium on Spatial user interaction (pp. 2-12). ACM.

[17] Feiner, S., MacIntyre, B., Haupt, M., & Solomon, E. (1993, December). Windows on the world: 2D windows for 3D augmented reality. In Proceedings of the 6th annual ACM symposium on User interface software and technology (pp. 145-155). ACM.

[18] Gebhardt, S., Pick, S., Leithold, F., Hentschel, B., & Kuhlen, T. (2013). Extended pie menus for immersive virtual environments. Visualization and Computer Graphics, IEEE Transactions on, 19(4), 644-651.

[19] Ha, T., Feiner, S., & Woo, W. (2014, September). WeARHand: Head-worn, RGB-D camera-based, bare-hand user interface with visually enhanced depth perception. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on (pp. 219-228). IEEE.

[20] Haas, C. (1992). Writing technology: Studies on the materiality of literacy. Mahwah, NJ: Erlbaum.

[21] Hutchings, D. R., Smith, G., Meyers, B., Czerwinski, M., & Robertson, G. (2004, May). Display space usage and window management operation comparisons between single monitor and multiple monitor users. In Proceedings of the working conference on Advanced visual interfaces (pp. 32-39). ACM.

[22] Jeon, M., Landry, S., Ryan, J., & Walker, J. (2014). Technologies expand aesthetic dimensions: Visualization and sonification of embodied Penwald drawings. In Proceedings of the 4th International Conference on Arts and Technology (ArtsIT2014).

[23] Jeon, M., Smith, M., Walker, J., & Kuhl, S. (2014). Constructing the Immersive Interactive Sonification Platform (iISoP). In HCI International.

[24] Kuhl, S.A., Thompson, W.B., & Creem-Regehr, S.H. (2006). Minification influenes spatial judgments in virtual environments. In Proceedings of the ACM SIGGRAPH Symposium on Applied Perception in Graphics and Visualization. ACM, New York, 15-19.

[25] Kuhl, S.A., Thompson, W.B., & Creem-Regehr, S.H. (2009). HMD calibration and its effects on distance judgments. ACM Transactions on Applied Perception 6, 3, 19:1-19:20.

[26] Kunz, B.R., Wouters, L., Smith, D., Thompson, W.B., & Creem-Regehr, S.H. (2009). Revisiting the effect of quality of graphics on distance judgments in virtual environments: A comparison of verbal

20

reports and blind walking. Attention, Percept., Psychophys. 71, 6, 1284-1293.

[27] Liu, Z. (2005). Reading behavior in the digital environment: Changes in reading behavior over the past ten years. Journal of documentation, 61(6), 700-712.

[28] Lovelace, E.A. and Southall, S.D. (1983). Memory for words in prose and their locations on the page. Memory and Cognition, 11, 429-434.

[29] Ma, J., Walker, J., Wang, C., Kuhl, S.A., & Shene, C.K. (2014). FlowTour: An Automatic Guide for Exploring Internal Flow Features. IEEE PacificVis.

[30] Mohler, B.J., Creem-Regehr, S.H., & Thompson, W.B. (2006). The influence of feedback on egocentric distance judgments in real and virtual environments. In Proceedings of the Symposium on Applied Perception in Graphics and Visualization. ACM, New York, 9-14.

[31] Monsell, S. (2003). Task switching. Trends in cognitive sciences, 7(3), 134-140.

[32] Nan, X., Zhang, Z., Zhang, N., Guo, F., He, Y., & Guan, L. (2014). vDesign: a CAVE-based virtual design environment using hand interactions. Journal on Multimodal User Interfaces, 8(4), 367-379.

[33] O'Hara, K., & Sellen, A. (1997, March). A comparison of reading paper and on-line documents. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (pp. 335-342). ACM.

[34] Reiling, F. (2014). Toward General Purpose 3D User Interfaces: Extending Windowing Systems to Three Dimensions. (Master's Thesis). Retrieved from https://github.com/evil0sheep/MastersThesis/blob/master/thesis.pdf?raw=true

[35] Robertson, G., Czerwinski, M., Baudisch, P., Meyers, B., Robbins, D., Smith, G., & Tan, D. (2005). The large-display user experience. Computer Graphics and Applications, IEEE, 25(4), 44-51.

[36] Robertson, G., Van Dantzich, M., Robbins, D., Czerwinski, M., Hinckley, K., Risden, K., Thiel, D., & Gorokhovsky, V. (2000, April). The Task Gallery: a 3D window manager. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 494-501). ACM.

[37] Sahm, C.S., Creem-Regehr, S.H., Thompson, W.B., & Willemsen, P. (2005). Throwing versus walking as indicators of distance perception in real and virtual environments. ACM Transactions on Applied Perception 1, 3, 35-45.

[38] Seward, Z.M. (June, 2014). Virtual reality headset Oculus Rift meets the Bloomberg terminal. Quartz. http://qz.com/218129/virtual-reality-headset-oculus-rift-meets-the-bloomberg-terminal/

[39] Simmons, T., & Manahan, M. (1999, January). The effects of monitor size on user performance and preference. In Human Factors and Ergonomics Society Annual Meeting Proceedings (Vol. 43, No. 24, pp. 1393-1393).

[40] Sublime Text. http://www.sublimetext.com/

21

[41] Steinicke, F., & Bruder, G. (2014, October). A self-experimentation report about long-term use of fully-immersive technology. In Proceedings of the 2nd ACM symposium on Spatial user interaction (pp. 66-69). ACM.

[42] Topol, A. (2000, April). Immersion of Xwindow applications into a 3D workbench. In CHI'00 extended abstracts on Human factors in computing systems (pp. 355-356). ACM.

[43] Trubek, A. (August 15, 2011). Out of Touch with Typing. MIT Technology Review. Retrieved from http://www.technologyreview.com/view/425018/out-of-touch-with-typing/

[44] Walker, J., Ma, J., Kuhl, S.A., & Wang, C. (2013). An Evaluation of Flow Field Visualizations with Internal Views. ACM Symposium on Applied Perception. Poster.

[45] Walker, J., Zhang, R., Kuhl, S.A. (2012). Minification and gap affordances in head-mounted displays. ACM Symposium on Applied Perception. Poster.

[46] Wayland. http://wayland.freedesktop.org/

[47] Wayland 3D Compositor on Oculus Rift /r/linux (Wayland 3D Compositor on Oculus Rift : linux) http://www.reddit.com/r/linux/comments/287nup/wayland_3d_compositor_on_oculus_rift/

[48] Williams, B., Narasimham, G., Westerman, C., Rieser, J., & Bodenheimer, B. (2007). Functional similarities in spatial representations between real and virtual environments. ACM Transactions on Applied Perception (TAP), 4(2), 12.

[49] Williams, B., Rasor, T., & Narasimham, G. (2009). Distance perception in virtual environments: A closer look at the horizon and the error. In Proceedings of the 6th Symposium on Applied Perception in Graphics and Visualization. ACM, New York, 7-10.

[50] Yost, B., Haciahmetoglu, Y., & North, C. (2007, April). Beyond visual acuity: the perceptual scalability of information visualizations for large displays. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 101-110). ACM.

[51] Zhang, R., Nordman, A., Walker, J., & Kuhl, S.A. (2012). Minification affects verbal- and action-based distance judgments differently in head-mounted displays. ACM Transactions on Applied Perception.

22