3D and Immersive Interfaces

3D Interfaces – Shneiderman

• An “extreme” interpretation of dm in which interfaces approach richness of real 3D world is wrong … to paraphrase Shneiderman

• In fact 3D interfaces are most often not right, even for tasks for which they may seem appropriate

– E.g., air traffic control

• When used 3D interfaces have a daunting set of challenges– E.g., occlusion, navigation in 3 space– E.g., “cone tree” rep. of file structure– So …

3D Interfaces – Shneiderman

• An “extreme” interpretation of dm in which interfaces approach richness of real 3D world is wrong

• In fact 3D interfaces are most often not right, even for tasks for which they may seem appropriate

– E.g., air traffic control

• When used 3D interfaces have a daunting set of challenges– E.g., occlusion, navigation in 3 space– E.g., “cone tree” rep. of file structure– So, …

Introduction

• The “best” interfaces, and all systems, typically find their task utility through engagement (etc.) appropriate for the task

– This idea is at the core of arguments for the use of direct manipulation interfaces

• All of the following are interrelated:– Immersion, engagement, presence, virtual reality– 3D display and interaction devices

• In field of CS and HCI: “spatial interfaces” and “immersive interfaces”

• Also, will introduce the idea of presence

Examples of Immersive Interfaces

Tiled display wall

Head mounted display

Surround screen projection

Spherical projection

Immersion, “Virtual Reality”, and Virtual Environments

• Immersive interfaces– High sensory immersion – visual, auditory, haptic, proprioceptive

• “Virtual reality”, or, virtual environments– “Virtual reality is a technology that is used to generate a simulated environment

in digital form... Using the equipment, users are immersed in a totally virtual world.”

– Working definition – an immersive interactive system

• In context of “virtual reality”, immersion usually = spatial immersion

• Note: “Immersion” (and engagement and presence) is a continuum

– Text ... Visual and 3d .. Stereo ... HMD… “jacked in”– Cyberspace

• Term coined by Gibson in Neuromancer• … and in the 21st century, the Matrix

Immersion and Virtual Reality

• “The mind has a strong desire to believe that the world it perceives is real” – Jaron Lanier, among others

• For example, “illusion” (perception) of depth (for spatial immersion)• Stereo parallax• Head motion parallax• Object motion parallax• Texture scale

• Interaction: grab and move an object

• Proprioceptive cues: – when you reach out and see a hand where you believe your hand to be,

you accept the hand as your own

• Often you will accept what you see as “real” even if graphics poor

• Constellation of cues

Components of Spatial ImmersionCutting, 1996

• Perception of 3 dims strongest element of spatial immersion

– Perception of 3d from depth cues

• See figure

• Other elements– Integration important– Visual display types– Stereoscopic display – Head position sensing – Hand-position sensing – Force feedback – Sound input and

output – Other sensations

Presence “The Aesthetic Impression of 3D Space”

• Sense of presence – Vividly 3d– Actually present in the world– Sense of being there– Holodeck …

• Presence has to do as much with engagement, as visual information– E.g., one can be “in the world”, when reading– Here, one sees, or visualizes, the world

• 3D depth cues are those elements that enhance feeling of 3 (vs. 2) dimensions in a display, – From occlusion to stereoscopic display

Presence “The Aesthetic Impression of 3D Space”

• Immersive interfaces– term used to describe interfaces/devices which lead toward immersion

(sense of presence, engagement) in the virtual environment presented on the display

• Virtual reality interfaces– term used similarly to immersive interfaces

• Degree of immersion– conventional desktop screen– fishtank virtual reality (semi-immersive workbench)– immersive virtual reality– augmented reality with video or optical blending– … number of cues …

Pictorial Depth CuesWhere does perception of three dimensions come from?

• 3D depth cues – Static / pictorial vs. dynamic– Monocular vs. binocular– Oculomotor

• Static monocular cues– Occlusion– Relative size– Linear perspective– Texture gradient– Aerial perspective– Shading– Relative height

• Courses in computer graphics and visualization provide detail

History - Sutherland’s Sketchpad“In the beginning …”

• Ivan Sutherland– “Pioneer” of … lots of things– Visualization– Graphics– Interaction– Still around

• Evans and Sutherland graphics

• First truly interactive graphics system, Sketchpad

– A fairly sophisticated “paint” (or drawing) program

• MIT, Ivan Sutherland’s 1963 Ph.D. thesis

– “Sketchpad, A Man-Machine Graphical Communication System”

• Available: www.cl.cam.ac.uk/techreports/UCAM-CL-TR-574.pdf

• Video: www.youtube.com/watch?v=mOZqRJzE8xg

• Among most important works in computer science

Ivan Sutherland using Sketchpad in 1963 CRT monitor, light pen and function-key panel

Ivan Sutherland’s Sketchpad, 1963and “the ultimate display”

• Regarded as the first to implement much of what called “hci”, “visualization”, “immersion”, and “virtual reality” (not to mention cg)

• Some quotes:

– ….. If the task of the display is to serve as a looking-glass into the mathematical wonderland constructed in computer memory, it should serve as many senses as possible.

– ….. By working with such displays of mathematical phenomena we can learn to know them as well as we know our own natural world. Such knowledge is the major promiseof computer displays.

– ….. The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked.

Sensorama, 1965Less Profound maybe, but Fun –

• Morton Heilig– cinematographer / director of

documentaries

• Motorcycle simulator - all senses– visual (city scenes)– sound (engine, city sounds)– vibration (engine)– smell (exhaust, food)

• Not a big commercial success, but “immersive”

from Virtual Reality Technology, Burdea & Coiffet

USAF Super Cockpit, 1985

• Wright Patterson Air Force Base

• Visual, auditory, tactile

• Head, eye, speech, and hand input

• Designed to deal with problem of pilot information overload– Flight controls and tasks too

complicated

• Research only– big system, not safe for ejecting

Immersive and 3D Interfaces

• Teleoperation

• Virtual and augmented reality

• Immersion and VR – contribution of components …

• Survey of 3D displays– Surround screen displays - CAVE– Input devices - Data glove– Data walls– Workbenches– Hemispherical display– Head-mounted displays– Arm-mounted displays– Virtual retinal display– Autostereoscopic displays

Immersive and 3D Interfaces

• Degree of immersion– conventional desktop screen

• Some 3D displays:– Data walls– Workbenches– Hemispherical display– Head-mounted displays– Arm-mounted displays– Surround screen displays - CAVE– Virtual retinal display– Autostereoscopic displays

• But first, … about stereoscopic display– A “compelling” 3D depth cue

• Is that part of human computer interaction– Yes

3D Interfaces, Stereopsis“Discovery” of Stereopsis, 1838

• Charles Wheatstone– Prolific scientist, Wheatstone bridge

• “… the mind perceives an object of three-dimensions by means of the two dissimilar pictures projected by it on the two retinae…”, 1838

• Contributions to the Physiology of Vision.—Part the First. On some remarkable, and hitherto unobserved, Phenomena of Binocular Vision.

• "Philosophical Transactions" of the Royal Society of London, Vol. 128, pp. 371 – 394, 1838.

• http://www.stereoscopy.com/library/wheatstone-paper1838.html

What the left and right eye see (retinal image)

Getting Different Images to Each Eye Wheatstone’s Stereoscope

• “The stereoscope is represented by figs. 8. and 9; the former being a front view, and the latter a plan of the instrument. A A' are two plane mirrors, about four inches square, inserted in frames, …”

• Below, modern mirror stereoscope using computer monitors

Getting Different Images to Each Eyethese days

• Commodity televisions– 120 hz display rate– Alternately, left then right

eye image– Active lcd glasses,

alternately block left eye, then right eye image

• lcd lens is alternately transparent, then opague

• 60 hz left eye, • 60 hz right eye

• Theaters– Polarized glasses

Getting Different Images to Each Eye Sutherland’s 1960’s equipment

• “Ultimate display”, 1965

• Sword of Damocles – 1st HMD– Actual camera-like shutters– Actual camera-like metal shutters

Visual Displays for VEs(Bowman)

• Types:– Standard monitor (mono/stereo)– Head-mounted/head-referenced– Projected (usually stereo)

• single-screen• multiple, surrounding screens

– Retinal display– Volumetric displays

• Characteristics of visual displays– Field of regard (FOR), field of view (FOV)– Brightness, contrast ratio– Resolution (two definitions)– Screen geometry– Light transfer– Refresh rate– Ergonomics

Remote (or tele-) Operation

• Combines:– direct manipulation in personal

computers– process control in complex

environments

• Physical operation is remote– Submarines, rovers, operating

rooms

• Complicating factors in architecture of remote environments:

– Time delays • transmission delays

• operation delays – Incomplete feedback – Feedback from multiple sources – Unanticipated interferences

Virtual and Augmented Reality

• Augmented reality shows real world with an overlay of additional overlay

• Knowlton (1975)

• Partially-silvered mirror over keyboard

• Programmable labels

• Tactile feedback

Augmented Reality, 1

• Enables users to see real world with an overlay of additional interaction– Situational awareness

• Typically, add text+images to real world

• See through glasses

• Very sensitive to head tracking, when used

Augmented Reality, 2

• Enables users to see real world with an overlay of additional interaction– Situational awareness

• See through glasses

• Typically, add text+images to real world

• Very sensitive to head tracking, when used

Augmented Reality, 3

• Heads up displays• Wearable displays - military applications

Augmented Reality, 4

• Heads up displays• Wearable displays - general use - Microvision

Surround-screen displays

• Pro• less obtrusive headgear• multi-user?• better stereo• Con• occlusion problem• missing sides

Immersive 3D DisplaysSurround Screen Systems

• Essentially same multi-surface display and interaction paradigm used today as 1992

• Orders of magnitude less hardware and software cost

Immersive 3D Displays Surround Screen Systems

• Essentially same multi-surface display and interaction paradigm used today as 1992

• Orders of magnitude less hardware and software cost

Computer(s)

UserTracking

Surface &Illumination1

Surface &Illuminationn

Software - modeling, …

.

.

.

Cruz-Neira et al., 1992

Surround screen displays – CAVE, 1

• A room with walls and/or floor formed by rear projection screens– Head tracking– Stereo– Light scattering

problems

• Visual immersion– Field of view is

100% possible, ~200 degrees

Surround screen displays – CAVE, 2

• Typical size: 10’ x 10’ x 10’ room

• 2 or 3 walls are rear projection screens– Floor is projected from above

• User is – tracked – He/she also wears stereo shutter goggles…– Uses “wand” to manipulate

• Projects 3D scenes for viewer’s point of view on walls

– Walls vanish, user perceives full 3D scene– So, view is only correct for that viewer

• Cost is fairly high

UTPA Immersive Systems Lab~Spring, 2014

Proj.

Proj. Proj.

27’13’

Security area

21’

6’

TV

stor

age

~8’

CAVE

Ph

ysio

log

ica

lM

easu

rem

ent

E

qu

ipm

ent

Computers

Front Projection Screen

Proj.

Develo

pmen

t

3D Input Devices, 1

• 3D input is hard

• Electromagnetic trackers– 6 DOF (position and

orientation in space)– can be attached to any head,

hands, joints• must deal with noise,

calibration– Polhemus tracker

• Optical trackers– photogrammetric technique:

space-resection by collinearity– no EM interference to worry

about– self-calibration– UNC’s Highball

3D Input Devices, 2

• Gloves – “gesture recognition”– attach electromagnetic tracker to

the hand– “breaks” fiberoptics

• Pinch gloves– contact between digits is a

“pinch” gesture

• Mouselike– 6 DOF– Logicad Magellan controller

Tactile Feedback

• Another avenue

“Data Walls”

• Very widespread use– Literature of practical

use

• Can use commodity projector

– E.g., with 27 as 3x9, with 1kx1k each gives

– 9,000 x 3,000

Workbenches, 1

• “One wall of CAVE” – rear stereo projection – fishtank view

• UNC NanoManipulator– Below with force feedback to “feel” carbon nanotubes with Atomic Force Microscope

Workbenches, 2

• Immersadesk is best know

• Tabletop displays• Pros

– direct manipulation– “god’s-eye” view– change orientation

• Cons– cancellation problem– small FOR

• Virtual retinal display (VRD)– HIT lab / Microvision– image scanned– directly onto retina great potential

Hemispherical display

• As with mirror stereoscope, high resolution possible

Head Mounted Displays, 1

• HMDs– Relatively high field of view (fov)– 90o direct FOV, 140o corneal FOV

• LEEP Optics (1975)– Large Expanse, Extra Perspective (LEEP)– Eric Howlett (Pop-Optix Labs)– Originally for 3D still photo viewing– Reported great realism for still images– Lenses correct for camera distortion

• Display optics matched to camera optics• Often uncorrected distortion for CG images

• And, more current technology:

Head Mounted Displays, 2

• HMDs– Relatively high field of view (fov), ~ 140 x 60

• NASA Ames HMD (1981-1984)

• McGreevy and Humphries– First implemented immersive HMDs– LCD “Watchman” displays

• NASA Ames VIEW or VIVID (1985)

• Virtual Interface Environment Workstation– Polhemus tracker, LEEP-based HMD, 3D audio,

Crystal River’s Convolvotron, Gesture recognition w/ VPL DataGlove, BOOM-mounted CRT (Sterling Software), Remote Camera (Fake Space)

HMDs now

http://www.vrealities.com/proviews035.html

DISPLAY Image Source Type: Single Full-color AMOLED (800x3 pixels) x 600 linesBrightness: 25 foot lamberts Contrast: >50:1

OPTICAL Field of View: 32° Horizontal x24° Vertical (40° Diagonal) @ SVGA resolution (800x600)26° Horiztonal x 19° Vertical (32° Diagonal) @ VGA resolution (640x480)Transmission: Non see-throughOptics: Plastic aspherical lensEye Relief: Eyeglasses compatible, >25 mmExit Pupil: Non-pupil forming

MECHANICAL Mounting: Standard: Clip on to Mich helmet* (Display module); Clip on to belt (Disply Controller)Weight: Display Module: 67 grams (w/out mount); Helmet Mount: 79 grams; Display Controller: 200 grams

DIMENSIONS Display Module (less mount): Height: 0.7"; Width: 1.7"; Depth: 1.2">br> Electronics Box: Height: 3.0"; Width: 2.8"; Depth: 1.2" Cable: Length: 1.2 meters

INTERFACE Plugable Display Module: Display Module includes 51" cable (Interface cable available separately) with water-resistant connectorDisplay Controller: Lemo Connector

ELECTRONIC Analog Video Rates: 800 x600 (SVGA 56 Hz to 85 Hz), 640x480 (VGA 60 Hz to 85 Hz)Analog Video Inputs: NTSC/RS-170 (A); CVBS and Y/C; R,G,B,HS,VSPower: System (VIN 6-24 VDC, 2.5 W nominal)Controls: RS-232 brightness and channel selection

DESIGNED TO MEET THE FOLLOWING PARAMETERS Temperature: Operating: -32° to +55°C; Storage: -32°C to +71°CHumidity: Six 48-hour cycles, 20°C to 55°C, 95% RHSalt Fog: Four 24-hour cycles (two wet, two dry)Vibration: Random vibration, 6 axis, 5 Hz to 2500Hz, up to 40 gsImmersion: Immersion in 1 meter of water for 2 hours

VR Price: $9,995.00

Add a Head Tracker! CLICK HERE Integrated Microphone with Voice Recognition Software Add $125.00

Wanting another payment option? Questions on products? Need help? CLICK HERE

http://www.cwonline.com/store/view_product.asp?Product=1151

• $799.00In Stock The new i-glasses PC models utilize LCOS micro-displays from Brillian Corporation with 800 x 600 resolution. These displays have been designed to achieve a much greater range of contrast and brightness capability. This results in an image substantially more vibrant and vivid. The new models will also reach their optimal operating temperature much faster and offer a very consistent contrast and color fidelity over the entire display. With built on integrated headphones. The Ultimate Portable PC Video System. Available in 3D model too! 

• SpecificationsResolution: 800x600

• Field of View: 26 Degrees Diagonal Image • Size: 76" at 13 Color Depth: 24 Bit Input • IPD Adjustments: None Required • Focus: 13 TBR • Eye Relief: 25mm • Exit Pupil: 17mmH x 6mmV • Convergence: 710", 100% Overlap, TBR • VGA / SVGA / XVGA Input: Scaled to SVGA (800 x

600)• Refresh Rate: Flicker Free 100Hz • Audio: Full Stereo Weight: 7 Ounces • Adjustments: Adjusts to fit all individuals • Control Features: On / Off, Volume Control • Power Supply: 100-240V AC Power Cube • Audio Connector: 1/8" Power: Barrel Connector

http://www.vrealities.com/addvisor150.html

• The AddVisor 150 personal display system is designed for the high-end professional market for Head-Mounted Displays with AR and VR capability. The AddVisor 150 is a transparent high-resolution Head-Mounted Display. The image is generated on two independent full colour 1280x1024 pixels (SXGA) Liquid Crystal on Silicon microdisplays. A light weight solution for the most demanding of applications the AddVisor 150 is recognized for excellent image quality, high brightness and contrast. The AddVisor 150 is ideal for direct view mono- and stereo (3D) viewing.

• The image can be shown superimposed on the environment with up to 35 % see-through or fully immersed. The AddVisor 150 is designed for a 46 degree diagonal 100% overlap field of view. A 50% overlap can also be used, giving a 54 degree horizontal or 60 degree diagonal field of view. It features a patented optical design that combines a wide field of view with high transparency see-through. Using a patent pending head fitting system with easy adjustments, low weight and eyeglass compatibility, the AddVisor 150 provides hours of easy and comfortable viewing.

• Specifications: • Resolution: 1280 x 1024 - SXGA

Field of View: 60 Degrees DiagonalImage Size: 76" at 13' Color Depth: 24 Bit InputIPD Adjustments: None RequiredEye Relief: Eyeglass compatibleConvergence: 7'10", 100% Overlap, TBRAudio: OptionalWeight: < 1 kgAdjusts to Fit all IndividualsControl Features: On / Off, Volume ControlPower Supply: 100-240V AC Power CubePower: Barrel Connector

• Call or E-mail for Price

Stereoscopic Viewing(Bowman)

• Stereopsis– Static, binocular cue– Each eye gets a slightly different image– Only effective within a few feet of viewer– Many implementation schemes

• Motion parallax– Dynamic, monocular cue– Near objects move faster than far objects– Generally more important than stereo!

• Oculomotor cues– Based on information from muscles in the Eye– Accommodation: lens shape (depth of focus)– (Con)vergence: gaze direction

Fyi - Stereoscopic Viewing(Bowman)

• Accommodation-convergence mismatch– Standard stereo displays confuse the brain

based on oculomotor cues– Only “true 3D” displays can provide these

LCD Shutter Glasses

• How to show different images to each eye

• Different images on odd and even– >120 hz monitor refresh

• LCD “Shutters” open and close for left and right eyes

– Recall, metal shutters

• From Stereographics web site:– Weight: 3.3 oz. (93 grams)– Shutters: Liquid Crystal– Field Rate: From 80 to 160 fields per second– Transmittance: 32% typical– Dynamic Range: 1500:1 typical– Battery Life: >250 hours of continuous operation– Battery Type: Two 3V lithium/manganese dioxide– Emitter

• Designed for PC/Unix desktop workstations using 3 pin mini-DIN connectors.

• Emitter for workstations allowing control over the IR spread (for multi-user environments).

• Connectors: 3-pin mini-DIN Right!

“Fish Tank VR” / Immersion

• Stereoscopic viewing– Depth of image appears to

be from about 12” behind and 6” in front of display

– Hence, a fish tank

• Often with “head coupled display”– Position of head is tracked– Image changed to appear

as if “looking around”

“Arm Coupled” Display with Boom

• Variation

Virtual Retinal Display

• Eric Seibel, U. Washington Human Interface Technology Lab– http://www.hitl.washington.edu/research/vrd/– www.mvis.com (commercial version)

• Simple enough: shine a laser in your eye and modulate it real fast

• Potential for wearable very high resolution virtual or augmented reality

Virtual Retinal Display

• “Paint” all of retina for immersive display, or part for augmented

• Monochrome 1000x1000– Color 3 lights

End?

Evaluation(Bowman)

• Case studies - Silva• Evaluate a new bodybased

interaction technique for the desktop game World of Warcraft

• Interviews• Think aloud• Critical incidents• Suggestions for improvement• Silva, M. and Bowman, D.

Body-based interaction for desktop games. Submitted to CHI 2009 Workin-Progress.

Which Visual Display to Use?(Bowman, 2004)

• Consider lists of pros and cons

• Consider depth cues supported

• Consider level of visual immersion

• Hhard question to answer empirically

• Instead of comparing actual displays, compare levels of immersion

Visual Display Types Comparison(Bowman, 2004)

Visual Display Types Comparison(Bowman, 2004)

End

• .