Interaction in the Virtual World: Overview• User interface• Navigation

– Wayfinding– Travel

• Interaction and communication with others– Sharing– Collaboration

• Manipulation– Methods– Properties– Selection– Operations

Broad Classification of Interactions

• Interactions– Reality-based– Imagination-based (magic)

Navigation

• Types of navigation goals: explore rate or position controlled), search (target) and maneuver (objects, environment)

• Methods:– Wayfinding– Travel

Navigation: Wayfinding (knowing where you are and where you’re going)

• Need a mental or cognitive map

• Relationship to the world, to the destination

• Situational awareness

• eg. of GPS system maps

• Great differences in people, sometimes gender related

• Problem of becoming lost, metaphors

Navigation: Styles

• Relative motion (steps)

• Absolute motion (point to a location and go)

• Teleport

• Move the world instead of the user

Navigation: Wayfinding (con’t)

• Mental map:– Divide and conquer: small regions, then piece

together– Global network: use landmarks– Memorize a map of the space– Remember a story about the space

Navigation: Wayfinding (con’t)

• Wayfinding aids– Path following: color, lines, labels– Maps (exo or ego): scale, icons, you-are-here– Landmarks: objects, audio– Placenames– Breadcrumbs (trail)– Compass– Instrument guidance: could be multimodal– Shift to exocentric view– Display of coordinates or grids, names– Constrain travel

Navigation: Travel Properties

• Manipulation method: physical (steering wheels, flight, dashboard) and virtual controls

• Constraints: terrain following, no z-axis, no lateral

• Frame of reference: relative motion

• Movement: gain, velocity, acc; voice

Navigation: Types of Travel

• Physical locomotion: direct tracking, issue of working volume

• Vehicle simulation

• Magic techniques, including steering, target-based travel

• Combination of natural and magic: manipulation-based travel

• Ride along or towrope: pre-set path

Navigation: Types of Travel (con’t)

• Fly-through, walkthrough, pilot-through: pointer directed, gaze-directed, torso-directed, dual-handed

• Move the world• Scale the world• Put me there, jump to destination• Time travel

Navigation and Travel

• Physical walking

• Vehicle simulation

• Magic techniques, including steering, target-based travel

• Combination of natural and magic: manipulation-based

Interaction: Communication

• Shared experience: same world, may be competing or co-existing

• Collaborative experience: need to work together in some way

Shared Experience

• Different ways of sharing in different mediums

• What can be shared: ideas, world, speech• Ways to share viewpoint

– Full multipresence: may have same equipment (such as HMDs), or be in same location (cockpit)

– Some viewers have partial presence: one immersed participant with onlookers, open display (possibly projection)

Shared Experience (con’t)

• Purpose of sharing: marketing (onlookers become interested), inc. throughput (onlookers get some experience even if not the full experience)

Collaborative Experience

• Communication: aural, visual, gestures, teleconferencing, marks or notes, physical proximity

• Synchronous communication: live conversations• Asynchronous communication: change a world

but then not be present, present in another part of the world – can leave marks or messages

• Can have both synchronous and asynchronous in the same world

Collaborative Experience (con’t)

• Messages or annotations: recipient, time (when needed), purpose– How do you know it’s there: voice, text,

gestures, pictures– Location– Point of view– Time– placeholders

Collaborative Experience (con’t)

• Who’s in charge (floor control and permissions): no one, moderated, permissions, formal, hierarchical, first come first served

Metacommands

• Manipulation of environment by someone other than participant (eg. of VR therapy)

• User can cause different worlds to load

Manipulation: methods

• Direct user control: gestures mimic real world

• Physical control devices: wands, buttons, joysticks, mouse

• Virtual controls: interfaces on the screen such as buttons, sliders, control panel, arrows, etc.

• Agent controls: intelligent intermediary

Manipulation: properties

• Feedback: user needs to know when something has happened- haptic (can feel the contact), aural (sound), visual (change color)

• Ratcheting: repeat input – eg. pick up mouse

• Constraints: restrict a DOF, snap to grid, lock to surface, keep on floor

Manipulation: properties (con’t)

• Distance: manipulate objects beyond reach

• Pointer beam scope: shape of beam; eg. laserbeam, spotlight, conelike

Manipulation: properties (con’t)

• Frame of reference: world representation, local representation– inside or outside the world- telepresence and teleoperation-egocentric or exocentric (god’s eye)-– Coordinate systems: world, local, parent-

vocabulary of DOF: vertical, lateral, longitudinal; Euler angles; yaw, pitch, roll

Manipulation: properties (con’t)

• Bimanual interface: sometimes with haptics, keypress with joystick (difficult), medical applications; must be well designed

• Control location: omnipresent or appear, placement (interfere with world or not): part of the world (door handle), attached to hand, front of the view (helmet, windshield), on the display, on a 2D panel of controls, physical devices (tablets)

Manipulation: properties (con’t)

• Control visibility: visible and invisible; how made visible

• Movement formula: gain (how much movement in one action), derivatives (velocity, acceleration)

Object Manipulation

• Manipulation of the environment: could also be considered viewpoint navigation

• Selection of objects: pointing, picking, grasping

• System control: usually through 2D interfaces, buttons

Manipulation: selection• Can be direction, item or value

• Direction– Pointer-directed: gesture– Gaze or eye tracking– Crosshair: pointer and gaze– Torso for direction of travel– Joystick, mouse, trackball- what about angles– Coordinates: possibly by voice– Landmarks

Manipulation: selection (con’t)• Item selection

– Contact between avatar and item – choose part of body; need feedback

– Point to select (prop, gesture)– 3D cursor select; may need button; question

of what’s in range and out of range; change in symbol

– Pinching or grasping– Naming– Menu; words or pictures– Select in a small world

Manipulation: selection (con’t)• Value selection: pen devices, keyboard,

tablet, physical sliders or controls, agents

Manipulation: operations

• Positioning and sizing objects

• Exerting force: pushing, hitting, supporting- does it move?

• Modifying attributes of objects: color, light, transparency, weight, shape, firmness, density, etc.

• Modifying global attributes: sounds, time of day, form of rendering

Interactions• Reality-based interaction:

– Head tracking, locomotion, 3D pointing– Direct manipulation of objects, gloves, haptics– Indirect manipulation through input devices– Objects for manipulation, tools (eg. wands), tangible

input devices, sliders, joysticks, touch sensors• Imagination-based interaction:

– Suspension of physics– Scaling of geometry, zooming, no clear standards yet– Scaling of motion– Automation of tasks, motion, etc.– Magic spells: dynamically assign meaning and tasks– Mode changes: eg from selecting to manipulation

SourcesBuilding on Realism and Magic for Designing 3D Interaction

Techniques by Kulik, IEEE Computer Graphics and Applications, Nov/Dec 2009

Questioning Naturalism in 3D User Interfaces, Bowman, McMahan, and Ragan, Comm. Of the ACM, 2012

Understanding Virtual Reality by Sherman & Craig, Morgan Kaufman, 2003

Computer Graphics and Virtual Environments by Slater et al