11

Tracking Systems in Tracking Systems in VRVR

22

Optical TrackersOptical Trackers Photo sensors detect a Photo sensors detect a

range of the range of the electromagnetic spectrumelectromagnetic spectrum– Usually arranged in 2D gridUsually arranged in 2D grid– CCD (old) or CMOS (fast) CCD (old) or CMOS (fast)

A single 2D image point A single 2D image point provides what information?provides what information?– LocationLocation– Color?Color?– IntensityIntensity

Camera = Sensor + Lens + Camera = Sensor + Lens + Color FilterColor Filter

X pixels

Y pixels

Simple Camera ModelSimple Camera Model You want a projection of the You want a projection of the

3D world3D world– i.e. what your eyes seei.e. what your eyes see– Sensor by itself does not do Sensor by itself does not do

thisthis

Pinhole camera is easy to Pinhole camera is easy to understandunderstand– Each pixel represents light Each pixel represents light

from a particular directionfrom a particular direction– Poor light gatheringPoor light gathering

Lens Lens approximatesapproximates pinhole pinhole– Focuses light from a Focuses light from a

particular point in spaceparticular point in space– Problem with depth-of-fieldProblem with depth-of-field– Lens distortionsLens distortions

Jan 9Jan 9 33

Simple Camera CalibrationSimple Camera Calibration Focal LengthFocal Length

– Distance of pinhole to image planeDistance of pinhole to image plane– Along with sensor size determines Along with sensor size determines

field of viewfield of view Center of projectionCenter of projection

– Was center of sensor glued directly Was center of sensor glued directly along optical axis?along optical axis?

– Can be assumed to be Can be assumed to be res_x/2,res_y/2res_x/2,res_y/2

Position and OrientationPosition and Orientation

Well known techniquesWell known techniques– ImagePoint= C*WorldPointImagePoint= C*WorldPoint– C is the camera matrix (3 x 4)C is the camera matrix (3 x 4)– 2 equations per world point, means 2 equations per world point, means

6 required points to obtain C6 required points to obtain C– In practice we use many moreIn practice we use many more– From C we can segment focal length, From C we can segment focal length,

center of projection, position and center of projection, position and orientationorientation

44

Issues in Camera TrackingIssues in Camera Tracking Isolating tracking pointsIsolating tracking points

– Which pixels contain the tracked object?Which pixels contain the tracked object?

Finding point correspondencesFinding point correspondences– Which pixels correspond to what on the Which pixels correspond to what on the

tracked objecttracked object

Mapping tracked object to 3D poseMapping tracked object to 3D pose

Camera resolutionCamera resolution– Limits accuracy and sensitivityLimits accuracy and sensitivity

Camera exposure timeCamera exposure time– Limits update rateLimits update rate

Points are rarely a single pixelPoints are rarely a single pixel– This can be a good thing (additional constraint, This can be a good thing (additional constraint,

sub-pixel accuracy)sub-pixel accuracy)

55

Optical Object TrackingOptical Object Tracking(with calibrated camera and known point correspondences)(with calibrated camera and known point correspondences)

1 point – direction of 1 point – direction of object relative to cameraobject relative to camera

2 points – direction of 2 points – direction of object, roll of cameraobject, roll of camera

3 non-colinear points – 6 3 non-colinear points – 6 degrees of freedom (4 degrees of freedom (4 possible solutions)possible solutions)

4 coplanar, non-colinear 4 coplanar, non-colinear points – 6 degrees of points – 6 degrees of freedom (unique solution)freedom (unique solution)

66

MarkersMarkers Pure image features (e.g. Pure image features (e.g.

corners) are difficult to use, corners) are difficult to use, segment, and if possible take segment, and if possible take too long to do (lag…)too long to do (lag…)

MarkersMarkers– Fast segmentation, high Fast segmentation, high

accuracy, point accuracy, point correspondencescorrespondences

– Encumbrance, occlusion, setupEncumbrance, occlusion, setup– Active Marker: marker emits Active Marker: marker emits

lightlight– Passive Marker: absorbs lightPassive Marker: absorbs light

Reflective markers are a good Reflective markers are a good compromisecompromise– Strobe light source, low-Strobe light source, low-

exposure, infra-red imagingexposure, infra-red imaging– Expensive camerasExpensive cameras 77

Optical Tracking Optical Tracking ConfigurationsConfigurations

Outside-looking-in Outside-looking-in (fixed camera (fixed camera positions, moving positions, moving objects)objects)

Inside-looking-out Inside-looking-out (fixed objects, (fixed objects, moving camera)moving camera)

1-N Cameras1-N Cameras

88

99

Multiple Camera TrackingMultiple Camera Tracking(with calibrated cameras and image correspondences)(with calibrated cameras and image correspondences)

Single point is a direction Single point is a direction (ray) from each camera(ray) from each camera

Known positions and Known positions and orientations of camerasorientations of cameras

Find shortest line segment Find shortest line segment between camera raysbetween camera rays– May not exactly intersectMay not exactly intersect

1 point provides position 1 point provides position onlyonly

Combine multiple points Combine multiple points for orientationfor orientation– Quality dependent on Quality dependent on

distance between pointsdistance between points– Good for inside looking outGood for inside looking out– Bad for outside looking inBad for outside looking in

Lateral Effect Photo DiodeLateral Effect Photo Diode Like a photocell, but current output Like a photocell, but current output

proportional to center of light intensity. proportional to center of light intensity. If the LEPD can only see a single light, it If the LEPD can only see a single light, it

immediately senses the position of the light immediately senses the position of the light in the imager (direction)in the imager (direction)– Multiple lights on at a time are a problemMultiple lights on at a time are a problem

Superior update rate and little processingSuperior update rate and little processing Near 0 latencyNear 0 latency Hi-Ball tracker Hi-Ball tracker

– 6 LEPDs6 LEPDs– Ceiling of LED strip lightsCeiling of LED strip lights– 1 light on at a time (fast sequence)1 light on at a time (fast sequence)– Best Best head-trackerhead-tracker in VR in VR

1010

Depth CamerasDepth Cameras In addition to measuring light intensity, also In addition to measuring light intensity, also

measure distance to light sourcemeasure distance to light source Two existing varietiesTwo existing varieties

– Time-of-Flight: Measures how long it takes for an Time-of-Flight: Measures how long it takes for an infrared light pulse to be picked up by each pixelinfrared light pulse to be picked up by each pixel High price / pixel, low latencyHigh price / pixel, low latency

– Structured Light: Measures properties of a Structured Light: Measures properties of a projected infrared light field (Kinect)projected infrared light field (Kinect) Low price / pixel, but “depth shadows”, processing Low price / pixel, but “depth shadows”, processing

latencylatency

1111

Tracking with Depth Tracking with Depth CamerasCameras

Big advantages over normal Big advantages over normal camerascameras– Simple background-foreground Simple background-foreground

segmentationsegmentation– Not dependent on image features for Not dependent on image features for

depth (unlike stereo camera pairs)depth (unlike stereo camera pairs)– Usually in addition-to, not in Usually in addition-to, not in

replacement of normal camerareplacement of normal camera Can track deformable bodiesCan track deformable bodies Known algorithms to fit articulated Known algorithms to fit articulated

body model to depth pointsbody model to depth points– Primesense NITE (used by FAAST)Primesense NITE (used by FAAST)

Based on calibration pose and best-fit Based on calibration pose and best-fit model (frame-to-frame) model (frame-to-frame)

– Microsoft Microsoft Based on machine learning algorithm to Based on machine learning algorithm to

recognize body partsrecognize body parts Depth is low precision right now, but Depth is low precision right now, but

will improvewill improve1212

1313

Optical Tracking ReviewOptical Tracking Review

Pros:Pros:– Can be inexpensiveCan be inexpensive– Wide areaWide area– Very accurateVery accurate– Wireless, near zero Wireless, near zero

massmass ConsCons

– High quality is very High quality is very expensiveexpensive

– OcclusionOcclusion– CalibrationCalibration

Data returned: 6 DOFData returned: 6 DOF Spatial distortion – very low (very Spatial distortion – very low (very

good accuracy)good accuracy) Resolution – very goodResolution – very good Jitter (precision) – very goodJitter (precision) – very good Drift - noneDrift - none Lag – moderateLag – moderate Update Rate – low - highUpdate Rate – low - high Range – very large (40’ x 40’ +)Range – very large (40’ x 40’ +) Number of Tracked Points – 4Number of Tracked Points – 4 Wireless - yesWireless - yes Interference and noise – occlusionInterference and noise – occlusion Mass, Inertia and Encumbrance - Mass, Inertia and Encumbrance -

moderatemoderate Price – cheap to very expensivePrice – cheap to very expensive

Optical Tracking Optical Tracking Performance Performance (Outside Looking In)(Outside Looking In)

1414

Variable Quality (Relative to alternatives)

Accuracy Excellent Position (<1mm), Good Orientation (<1 degree)

Resolution Excellent (<.1mm <.1degree)

Jitter Good position, Poor orientation (for small markers).

Drift None

Lag Okay (better with on board electronics)

Update Rate Poor –> Excellent (30hz– 1500hz)

Interference Okay (line of sight)

Encumbrance Very good position (low weight, wireless), okay orientation (large marker)

Space Very good (depends on camera distance), large physical space requirements

Tracked Entities Excellent (no practical limit)

Calibration Poor (shifts over time, by end-user)

Cost Get what you pay for ($$-$$$$-$$$$$$) High quality, fast cameras are very expensive

Optical Tracking Optical Tracking Performance Performance (Inside Looking Out)(Inside Looking Out)

1515

Variable Quality (Relative to alternatives)

Accuracy Okay Position (<1cm), Excellent Orientation (<.1 degree)

Resolution Excellent (<.1mm <.1degree)

Jitter Okay position, Excellent orientation.

Drift None-some (think optical mouse)

Lag Okay (better with on board electronics)

Update Rate Poor –> Excellent (30hz– 1500hz)

Interference Okay (line of sight)

Encumbrance Poor (cameras need a wire and have significant weight )

Space Very good (Easier to move than O-L-I), can track very large spaces)

Tracked Entities 1/ camera

Calibration Good (markers on walls tend to stay put).

Cost Get what you pay for ($$-$$$$-$$$$$) High quality, fast cameras are very expensive, but you only need to buy 1.

Hand/Finger TrackingHand/Finger Tracking

Special case of human body trackingSpecial case of human body tracking

Very difficult, why?Very difficult, why?– 14 joints in small area14 joints in small area

1616

Mechanical Data Mechanical Data GlovesGloves

Most common Most common solution to hand solution to hand trackingtracking

Measure joint anglesMeasure joint angles– Mechanical tracking Mechanical tracking

(but not good)(but not good) Encumbering, sweaty, Encumbering, sweaty,

still need to track still need to track hand, etc…hand, etc…

Need calibrationNeed calibration Poor performance in Poor performance in

generalgeneral Not user independentNot user independent

1717

Optically Tracked Data Optically Tracked Data GlovesGloves

Use specialized markers Use specialized markers to facilitate hand tracking to facilitate hand tracking in a small spacein a small space– Pulsed LEDs (A.R.T gmbh)Pulsed LEDs (A.R.T gmbh)– Colored Segments (MIT)Colored Segments (MIT)

1818