Thales Visionix, Inc. #AWE2016

1

Lessons Learned: Ten years of Motion Tracking for Mixed Reality applications

Tom CallowayAdvanced Programs Manager

Thales Visionix – Massachusetts, USA

Thales Visionix, Inc. #AWE2016 2

Outline

I. Motion Tracking for Mixed Reality

II. Lessons to Remember

III.The Future of Augmented Reality

Thales Visionix, Inc. #AWE2016 3

Motion Tracking: From Nice to Necessary

The utility of motion tracking has ranged from useful to absolutely necessary at points along the entire mixed

reality continuum

Thales Visionix, Inc. #AWE2016 4

Motion Tracking for Mixed Reality

“Tracking any combination of the degrees of freedom (DOF) can be useful for a given

application.

e.g. 1, AR for pilots (3DOF orient. or 6DOF pose)e.g. 2, VR (3DOF orient. or 6DOF pose)e.g. 3, Tracking a tool (3DOF pos. or 6DOF pose)e.g. 4, Tagging an object (6DOF pose + 1DOF range)e.g. 5, Deep see diving (1DOF depth)

Thales Visionix, Inc. #AWE2016 5

Motion Tracking Approaches are Diverse

All Kinds of Hardware: • Gyroscopes• Accelerometers• Ultrasonic transmitters and receivers• Magnetic sources and sensors• Cameras (mono, stereo, panoptic)• Depth sensors (structured light, time of flight)• Markers and fixed references (visible light,

reflective, LED)

All Kinds of Algorithms: • Kalman filtering (EKF, UKF, MSCKF)• ICP, RANSAC, particle filtering, smoothing• Image processing and computer vision

• Centroid refinement, natural feature detection, machine learning

• SLAM, VINS, VIO• Enhancement filters

• Prediction, virtual synchronization, perceptual enhancement

Thales Visionix, Inc. #AWE2016 6

Lessons to Remember

1.Understand your user and your application

2.Know your environment and your technology

3.Don’t settle for sub-optimal performanceUnless you truly have no choice

Thales Visionix, Inc. #AWE2016 7

Understand your User and your Application

Firefighters “kick” their way up stairs to test their stability and integrity.

Shoe mounted inertial tracking algorithm needs to be robust against, and even make use of that knowledge.

Thales Visionix, Inc. #AWE2016 8

Understand your User and your Application

Air Force pilots come in all sizes…

Thales Visionix, Inc. #AWE2016 9

Know your Environment and your Technology

Stinger Missile Dome Trainer with Wireless ultrasonic-inertial motion tracking system, Courtesy of AAI

Tracking systems fusing ultrasonic and inertial data can work extremely well in environments where other systems fail.

Doesn’t work so well in the presence of microphone-saturating explosion sounds though…

Thales Visionix, Inc. #AWE2016 10

Don’t Settle for Sub-optimal Performance

A tracking installation wasn’t proceeding optimally due to sighting equipment slipping in simulated Martian sand.

We switched to a fiducial auto-mapping approach that we use for aircraft cockpit augmented reality.

This prompted us to further improve our algorithms. We now get ~20 micron of RMS mapping precision for pilot augmented reality.

Thales Visionix, Inc. #AWE2016 11

So remember once again..

1.Understand your user and your application

2.Know your environment and your technology

3.Don’t settle for sub-optimal performanceUnless you truly have no choice

Thales Visionix, Inc. #AWE2016 12

The Future of Motion Tracking in AR

Enabling Augmented Reality Anywhere1. Inertial Sensors will become more accurate and dominate the approaches2. Computational power will continue improving (ASICs, 3D chips, local + cloud,

etc.)3. Utility of AR will expand in tandem with hardware and content4. Major software efforts including OS re-writes (ROS -> CyOS)5. Perfect storm of wearables, IoT, better algorithms, content, data and hardware6. The net result will be extremely positive, fun, and profitable!

Thales Visionix, Inc. #AWE2016 13

A glimpse of Keiichi Matsuda’s Hyper Reality dystopia

Thales Visionix, Inc. #AWE2016

14

Thank you!Come see the Thales Visionix booth in the AR Section of the Expo- See technology demonstrations (natural feature tracking and

AR)- Let us help you solve your motion tracking challenges