1

z

deti departamento de electrónica telecomunicações e informática universidade de aveiro

Introduction to VR and AR 3D Interfaces – Input

Paulo Dias

Outline

• Introduction to VR and AR

• 3D User interfaces – Input– Trackers

– Navigation and manipulation interfaces

– Gesture Interfaces

• Camera Calibration and Augmented Reality

• Practical labs: ARToolkit

2

Why Virtual Reality / 3D interfaces?

• 3D Digital information is common (3D reconstruction, games, CAD, Design,…)

• Interaction is still mainly 2D – keyboard and mouse from 70

• Virtual Reality: an attempt to go beyond 2D interaction and immerge user in 3D world

3

Apple II – from 1977

What is Virtual Reality

“A high-end user interface that involves real-time simulation and interaction through multiple sensorial channels.” (vision, sound, touch, smell, taste) (Burdea and Coiffet., 2003)

4

Virtual and Augmented Reality

5

Real

Environment

Augmented

Reality

Augmented Virtuality Virtual

Environment

Mixed Reality

Paul Milgram [Milgram et. al 1994]

Main challenges in VR/AR

• How to interact with users – Input Devices

• How to immerge user in the world – Output Devices

• =>No single solution: several trials with different setups

6

2

Virtual Reality example

• Equipment

– Tracker 3DOF (Yaw, Pitch, Roll)

– HMD SVGA estereoscópico

– WiiMote

– Wintracker 6DOF

• Graphical libraries

– VTK (Visualization Toolkit)

– OGRE

9

Interaction in RV

• Orientation– Yaw, Pitch e Roll from 3DOF tracker

– Direct mapping to camera

• Navigation– Acceleration sensor

– Movements map to actions

• Navigation mode– Axis, Point of view, etc…

10

Augmented Reality - equipment

• Equipment

– Virtual Reality + video camera

– Video see-through

• AR Toolkit library

– Marker detection

– Superposition of virtual elements in the world

• Virtual Environments in VRML

– Configuration file in txt

11

Results

12

Virtual and Augmented Reality Prototype

More recent setup

13

Input devices : Motion trackers

– Magnetic

– Mechanical

– Acoustic

– Optical

– Inertial

– Hybrids (combination of severals)

14

Polhemus Fast-Track

Boom

3D Mouse - Logitech

InertiaCube 3 - Intersense

Finger Tracking - ART

3

15

Input Devices Input devices

• Trackers:– Mechanical– Magnetic (AC, DC)– Optical– Ultrasonic– Inertial, – ...

• Navigation and manipulation interfaces:– Tracker-based– Trackballs– 3D mice, – ...

• Gesture interfaces:– Didjiglove– Cyberglove– ...

16

Virtual objects have 6 degreesof freedom (D.O.Fs):

-three translations;

-three rotations.

(Burdea and Coiffet., 2003)

Trackers measure the motion of “objects” (e.g. user head) in a fixed system of coordinates.

Tracker is a special purpose H/W to measure the real-time change in a 3D object position and orientation

Trackers

17

Without the head tracker - the image- the sound cannot change to match thehead posture

Required tracking accuracy: Image > sound

(Burdea and Coiffet., 2003)

Example magnetic in HMD

18

Body Tracking:

• Head

• Hand and fingers

• Torso

• Feet

• ...

Indirect tracking: Using physical objects (props and platforms)

Direct tracking: Tracking directly the body parts

What is usually tracked?

19

• Measurement rate – Readings/sec

• Sensing latency

• Sensor noise and drift

• Measurement accuracy (errors)

• Measurement repeatability

• Tethered or wireless

• Work envelope

• Sensing degradation

Trackers characteristics

20

4

• Accuracy

• Jitter

• Drift

• Latency

• Tracker update rate

Tracker performance parameters:

21

Tracker performance parameters should be analyzed to match a solution for sensorial channel and budget of an application

Real object position

Accuracy

Tracker position

measurements

Tracker characteristics

22

(Burdea and Coiffet., 2003)

Accuracy:Difference between the object’s actual 3D position and that reported by the measurement

Real object fixed

position

Signal noise

Time

Tracker data

Tracker characteristics

23

(Burdea and Coiffet., 2003)

Jitter:Change in tracker output when the tracked object is stationary

Real object fixed

position

Sensor drift

Time

Tracker data

Tracker characteristics

24

Drift: Steady increase in tracker error with time

(Burdea and Coiffet., 2003)

Real object

position

Sensor latency

Time

Tracker data

Tracker characteristics

25

Latency:Time delay between action and result: time between the change in object position/orientation and the time the sensor detects this change

(Burdea and Coiffet., 2003)

Tracker characteristics

26

(Burdea and Coiffet., 2003)

Tracker update rate:Number of measurements that the tracker reports every second

If the same tracker electronics is used to measure several objects, the sampling rate suffers due to multiplexing

Dedicated electronics to each tracked object

Same electronics to each tracked object

5

Mot used trackers

– Magnetic

– Mechanical

– Acoustic

– Optical

– Inertial

– Hybrids (combination of severals)

27

Polhemus Fast-Track

Boom

3D Mouse - Logitech

InertiaCube 3 - Intersense

Finger Tracking - ART

A mechanical tracker consists of a serial or parallel kinematic structure composed of links interconnected by sensorizedjoints.

Mechanical trackers

28

(Burdea and Coiffet., 2003)

Mechanical tracker - Push 1280 stereo display (Fakespace Inc)

Item is no longer available

Pros

• Use sensors imbedded in exoskeletons to measure position

• Have extremely low latencies

• Are immune to interference from magnetic fields and large

metal objects

Cons

• Limit the user’s freedom of motion

• Can be heavy if worn on the body

Mechanical trackers

29

Mechanical tracker - Example of an exoskeleton

30

http://www.youtube.com/watch?v=uJza6G-7tD4

A magnetic tracker is a non-contact position measurement device that uses a magnetic field produced by a stationary TRANSMITTER to determine the real-time position of a moving RECEIVER element

may be ACDC

31

Magnetic trackers

• Use low-frequency magnetic fields to measure position

• Fields are produced by a fixed source

• Size of source grows with the tracker work envelope

• The receiver is attached to the tracked object and has three perpendicular antennas

• Distance is inferred from the voltages induced in the antennas – needs calibration…

Magnetic trackers

32

6

Magnetic trackers with data glove

33

(Burdea and Coiffet., 2003)

Magnetic trackers widely used but expensive

34

Ascension

Tracking, guiding, and localizing medical instruments within a patient's body

3D measurement and analysis of human movement for biomechanical purposes: sports, performance, and design

Track head and objects for matching computer-generated imagery with head direction; weapon aiming and interactive instruction

Tracking of head, hands and 3D pointers for interaction with large scale and immersive displays

Wireless suit ascension technologies

35

Magnetic Tracker – Razor Hydra [low cost 79€]

36

• True six degree-of-freedom magnetic motion tracking

• Ultra precise sensor for 1mm and 1 degree tracking

• No line of sight to controllers required

• Low-power magnetic field

• low power consumption

• Low latency feedback

Razer Hydra

• Use mechanical measurements to reduce errors

• Sensor noise – variation in measurement with no

real object motion – solved by over-sampling

• Size of errors grow from source outwards

• Errors both in position and orientation

Magnetic trackers Calibration

38

Magnetic trackers accuracy degradation

39

(Burdea and Coiffet., 2003)

7

• DC trackers are immune to non-ferromagnetic metals

(brass, aluminum and stainless steel)

• Both DC and AC trackers are affected by the presence of ferromagnetic metals

(mild steel and ferrite)

• Both are affected by copper

• AC trackers have better resolution and accuracy

• AC trackers have slightly shorter range

DC vs AC magnetic trackers

40

A non-contact position measurement device that uses an ultrasonic signal produced by a stationary transmitter to determine the real-time position/ orientation of a moving receiver.

Ultrasonic trackers

41

(Burdea and Coiffet., 2003)

• Use low-frequency ultrasound to measure position

• Sound produced by a fixed triangular source (speakers)

• Number of sources grows with the tracker work envelope

• The receiver is triangular and attached to the tracked object and has

three microphones

• Distance is inferred from the sound time of flight

• Sensitive to air temperature and other noise sources

• Requires “direct line of sight”

• Slower than magnetic trackers (max 50 updates/sec)

Ultrasonic trackers

42

(Burdea and Coiffet., 2003)

http://inition.co.uk/3D-Technologies/logitech-head-tracker

6 DOFsUpdate rate - 50 / secLatency – 30 ms

Ultrasonic trackers - Logitech

A non-contact position measurement device that uses optical sensing to determine the real-time position/ orientation of an object

Optical trackers

44

(Burdea and Coiffet., 2003)

outside-looking-in inside-looking-out

LaserBIRD optical tracker

Optical trackers – outside looking in

45

(Burdea and Coiffet., 2003)

http://www.ascension-tech.com/realtime/laserbird2.php

Sub-degree and sub-millimeter accuracyMeasurement rate of 240 meas/secTracking response: 7.17 ms

8

Vicon MX

• Uses 4 Mpixel cameras with own 120 LED array (infrared, or visible red). Accuracy 0.02 of a pixel

• Camera has real-time onboard image processing (masking and thresholding)

• Resolution 2352x1728 @ 160 fps

• 8 cameras are connected to a MX net unit which then communicates with the PC

Optical trackers – outside looking in

47

(Burdea and Coiffet., 2003)

• MX Link connects several MX Net units for more cameras

• To interface with other devices like a force plate, sensing glove or eye tracker – use a MX control unit

• User wears reflective markers (small spheres)

(Burdea and Coiffet., 2003)

Optical trackers – outside looking in

• The best accuracy is close to the work envelope

• Very large tracking surface and resistance to visual occlusions (line of sight)

Inside-out optical tracker advantages/ disadvantages

49

(Burdea and Coiffet., 2003)

Inertial motion tracker

50

Xsens

Inertial trackers

• No interference from metallic objects

• No interference from magnetic fields

• Large-volume tracking

• “Source-less” orientation tracking

• Full-room tracking

Hybrid – Ultrasonic/Inertial trackers

51 52

Intersense IS 900

(Burdea and Coiffet., 2003)

Immune to metalic objects Accuracy: 2.0 - 3.0 mm Update Rate: 180 Hz Nominal

(120 Hz Nominal Wireless)Latency: 4 ms typical

http://www.intersense.com/pages/20/14

Inertial trackers

9

InterSense Stereo stylus tracker

Accelerometer

Ultrasonicemitter

53

http://www.mindflux.com.au/products/isense/is900.pdf

But…

• Accelerometer errors lead to decreased accuracy

• Errors grow with time!

• Gyroscope errors compound position errors

• Needs independent position estimation to reduce “drift”

54

Hybrid – Ultrasonic/Inertial trackers

InertiaCAMFixed-focus lens, on-board DSPWeight 35 gramsTracking range 10 m per 10 cm of fiducial diameter

Fiducials

• 6 DOF, accuracy 3 mm, 0.1 degree,

• Latency 2 ms;

• Requires a minimum of 1 fiducial

• for every 2 m2 at 2 m from camera;

• Update rate 120 Hz;

Hybrid vision - inertial

60

IS 1200

• Navigation interfaces allow relative position control of virtual objects

(including a virtual camera)

• Gesture interfaces allow dexterous control of virtual objects and interaction

through gesture recognition.

Navigation and Gesture Input Devices

61

• Trackballs

• 3D mice

• 3D probes

• Cubic Mouse

• Wiimote, …

• Perform relative position/velocity control of virtual objects

• Allow “fly-by” application by controlling a virtual camera

Navigation Input Devices

62 63

http://www.youtube.com/watch?v=1WuH7ezv_Gshttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.84.6127&rep=rep1&type=pdf

- Allows to intuitively specify 3D- coordinates

- The rods represent the X, Y, and Z axes of a given coordinate system

Cubic mouse

10

64

http://www.spacecontrol.us/3d-maeuse-spacecontroller-funktionsprinzip.htmlhttp://www.3dconnexion.com/products/what-is-a-3d-mouse.htmlhttp://www.youtube.com/3DMaus

• Sends relative coordinates

• Allows - to move 3D objects in a more intuitive way than a 2D mouse- allows rotation around each axis

• Price ~300 USD

3D mouse Space controller

http://www.axsotic.com/overview.html

- Has no mechanical sensors that can generate unwanted behaviors

- The specially developed polymer springbodies create a smoothly consistent workflow

3D-Spheric-Mouse

• Are sensing gloves such as:

- Fakespace Pinch Glove (switches)

- 5DT Data Glove (fiber-optic)

- DidjiGlove (capacitive)

- Immersion CyberGlove (stain gauges), …

• Have larger work envelope than trackballs/3-D probes

• Most need calibration for user’s hand

68

5DT Data Glove

DidjiGlove5DT Data Glove

CyberGlove

Pinch Glove

Gesture Input Devices

Finger Degrees of Freedom

69

70

(Burdea and Coiffet., 2003)

Hand work envelope vs. interface type

- No joint measures ; does not need user calibration

- Detects contact between any two fingers

- RS-232 protocol communicates - gesture information - delta time stamps

- Does not provide location in space

- A mount compatible with - Ascension - Polhemus(electromagnetic trackers)

71

The Pinch Glove (Fakespace Co.)

http://inition.co.uk/3D-Technologies/fakespace-labs-pinch-gloves

Pinch glove (Fakespace Co.)

11

The glove interface: A) five-sensor version; B) newer design

Price: ~900$

http://www.5dt.com/products/pdataglovemri.html

http://www.vrealities.com/products/data-gloves/5dt-data-glove-5-ultra-2

A) One optical fiber/finger Roll/pitch sensing 5DT Data Glove Ultra

100 datasets/sec, 12 bit A/D flexion resolution, wireless version transmits data at 30 m, needs calibration

73

5DT Data Glove (5Th Dimension Technologies)

75

(Burdea and Coiffet., 2003)

http://www.youtube.com/watch?v=RqOcuzRP1aQ#t=21

5DT Data Glove (5Th Dimension Technologies)

78

http://inition.co.uk/3D-Technologies/didjiglove

• Captures the position of fingers by means of 10 capacitive bend sensors

• Uses one flex sensor/finger

• Angles are obtained by measuring voltages

• Communicates via RS232

DidjigloveThe DG5 glove

• 100 gestures/sec

• Small latency (10 ms)

• Sensor resolution 10 bit (1024 points)

• Cannot measure individual joints

• Needs user specific calibration

• Less expensive (about $500)

http://www.vrealities.com/products/data-gloves/dg5-vhand-glove-3-0

79

• Uses 18-22 linear sensors – electrical strain gauges

• 112 gestures/sec “filtered”

• Angles are obtained by measuring voltages on a Wheatstone bridge

• Sensor resolution 0.5 degrees, but errors accumulate to the fingertip

• Sensor repeatability 1 degree

• Needs calibration when put on the hand;

• Expensive (about $10,000)

81

http://www.metamotion.com/hardware/motion-capture-hardware-gloves-Cybergloves.htm

http://www.cyberglovesystems.com/

The CyberGlove (Cyber Glove Systems)

82

http://www.cyberglovesystems.com/

The CyberGlove (Cyber Glove Systems)

12

Specifications Pinch Glove

5DT Data Glove DG5 Glove

DG5 VHand

CyberGlove

Number of

sensors

Sensor type

Record/sec

Sensor

resolution

Communication

rate

Wrist sensors

7/glove

(2 gloves)

Electrical

NA

1bit

(2 Points)

Wired

(19.2kb)

None

5 or 14/glove

(1 glove)

Fiber-optic

100(5DT 5W),

200(5DT 5)

8bit

(256 Points)

Wireless(9.600kb)

Wired(19.2kb)

Pitch

(5DT 5 model)

5 /glove

(1 gloves)

?(ink film)

100

25 (DG 5 VHand)

10 bit

(1024 points)

Wired (19.2kb)

Wireless (DG5 VHand)

Accelerometers

(DG5 Vhand)

18 or 22/glove

(1 glove)

Strain gauge

150(unfiltered)

112(filtered)

0.5°

Wired

(115kb)

Pitch and yaw

84

Low cost - Entertainment Driven

87

• Wii

• Kinect

• Playstation eye

• Leap

Wii

• Button

• Accelerometer

• Infrared Camera

• Feedback

– Visual

– Touch

– Sound

• Battery

• Communication

88

Playstation eye

• Optical tracking

89

Kinect – Structured Light

• Active – Infrared pattern

90

Main bibliography

- G. Burdea and P. Coiffet, Virtual Reality Technology, 2nd ed. Jonh Wiley and Sons, 2003

- Craig, A., Sherman, W., Will, J., Developing Virtual Reality Applications: Foundations of Effective Design, Morgan Kaufmann, 2009

-

- J. Vince, Introduction to Virtual Reality, Springer, 2004

92

13

Aknowledgement

The author of these slides is greteful to:

• Burdea and Coiffet for making available the slides supporting their book

• All colleagues and students that have contributed in any way

93