Join the conversation @ARealityEvent & #AWE2017

Exploring the interactive boundaries of VR

Beyond the Hands

Chris Wren

WHAT’S THIS TALK ABOUT?

•  Survey of Existing/Upcoming Sensory Tech for VR •  Showcase my work in this area / Lessons learned

• VR Avatars •  Full Body VR • Mixed Reality • Hand Tracking

• A little bit of VR Social • Where are we going?

WHO AM I

Chris Wren •  12 years professionally developing games (Microprose/EA/Namco) •  9 years instructing Game Design at George Mason •  I do AR/VR research/consulting in Virginia: www.wrenar.com •  I work with an amazing AR/VR company: www.notiontheory.com

MyFirstJob

“TheSims”I shipped 7 Sims titles in 4 years.

Warhammer:MarkofChaos

Inspirational signposts of what AR/VR interaction might be like

More real than real.

WHY ARE WE DOING THIS?

TAPPING THE SENSES

SIGHT

•  Higher FOV •  Better Pixel Density •  Better Lenses •  Higher Framerate •  Beyond the Visual Spectrum •  Transferring to AR

FUTURE SIGHT

•  Thermal •  IR •  Lightfield • Retinal Projection • Bypassing Optic Nerve?

SOUND

Ambisonics

TOUCH

Reach out, reach out and touch someone. – 1980’s ad campaign from AT&T

HAPTIC GLOVES

NOTION THEORY GLOVES

VR TOUCH

Haptic Fingertips

RAINBOW JELLIES AND LEAP MOTION

Synesthesia

MONSTER SHOP AND USENS

Maybe we don’t need controllers…

TACTICAL HAPTICS

Strafe Haptics

CORNELL ORGANIC ROBOTICS

OmniPulse

SOFT ROBOTICS AND VR

Future Peripherals

UNLIMITEDHANDS

H2L Inc (China)

MUSCLE STIMULATION

Hasso-Plattner-Institut in Potsdam

WHERE IS THIS HEADED?

VIRTUAL HANDSHAKES

THE SUITS The outside of the suit was covered with an elaborate exoskeleton, a network of artificial tendons and joints that could both sense and inhibit my movements. Built into the inside of the suit was a weblike network of miniature actuators that made contact with my skin every few centimeters. These could be activated in small or large groups for the purpose of tactile simulation—to make my skin feel things that weren’t really there. They could convincingly simulate the sensation of a tap on the shoulder, a kick to the shin, or a gunshot in the chest. -Ernest Cline, Ready Player One

TESLA SUIT

SKINTERFACE

Student Project from London’s Royal College of Art

SUBPAC

HARDLIGHT VR

NullSpace VR

WHERE IS THIS HEADED?

VIRTUAL HUGS

SMELL

OLFACTORY BULB

FEEL REAL

Heat, Wind + Smells

NOSULUS RIFT

Not all smells are welcome in VR

BRAIN HACKED COOKIES

University of Tokyo

MOVEMENT I spent the majority of my time in my Shaptic Technologies HC5000 fully adjustable haptic chair. It was suspended by two jointed robotic arms anchored to my apartment’s walls and ceiling. These arms could rotate the chair on all four axes, so when I was strapped in to it, the unit could flip, spin, or shake. -Ernest Cline, Ready Player One

CABLE ROBOTS

FraunHofer IPA

GIMBALS

TREADMILLS

VIRZOOM

MARS BUS

•  Made by Framestore

•  Uses Unreal Engine

•  Designed for real-world movement

RE-DIRECTED WALKING

VESTIBULAR STIMULATION

SCANNING

WALK AROUND 360 VIDEO

HYPE VR

Demoed by Intel at CES 2017

SHARED DATASETS

•  Think: Google Maps

•  Tracking Matrices

•  Only updating that which has changed

•  Confirming Data from multiple sources

BACKPACK VR

•  Allowed Freedom of Movement

•  Great with Redirected Walking

•  NOT Good for Athletics: Jumping

•  Made Layups, Running, Jumping Possible

•  Wireless non-backpack solution determined

VR AVATAR DEVELOPMENT

MY ORIGINAL VR AVATAR TEST

   

Kyle the Unity Robot(Just Head – No IK) - Oculus DK2 - May 2015

KYLE PLAYING BASKETBALL

   

Basketball Avatar Test – June 2016 (Steam VR IK Rigging)

KYLE JELLIES

   

Jelly Avatar Test – June 2016 (Unity IK Rigging)

THE UNCANNY VALLEY

Getting Away From The Robot

SCANNED-IN PEOPLE

   

Scanned Model from Sketchfab – Auto-rigged w/Mixamo (Unity IK Rigging)

HOVER JUNKERS IK

Using “Final IK” Rigging

SCANNED AVATAR + LEAP MOTION + FOOT CONTROLLERS

   

1st Foot Rigging - August 2016

FULL BODY VR

•  Running, Jumping in Real Environments

•  Interacting with real tracked objects

ENTER THE FEET

STRAP/CLIP SHOE EXPERIMENTS

OVERSHOE PUCK SOLUTIONS

SOCCER DEMO

ANOTHER MIXED REALITY DEMO

   

SOCIAL VR

MY VRML DAYS

   

ALTSPACE VR

   

BIGSCREEN VR

   

Hayden from (ConVRge)

VR CHAT

   

OCULUS SOCIAL

   

TENCENT VR HMD

Integration of WeChat – VR Social

V TIME

   

REC ROOM

   

PROJECT SANSAR

   

Linden Labs (Second Life)

HIGH FIDELITY

   

Philip Rosedale on stage Above

AR: EYE TRACKING

EYE TRACKING HMDS

SMI, Fove, Qualcomm

MAGIC LEAP

AR Lightfield

AVEGANT

First Standalone Headset to do Lightfield Synthesis

LOCATION BASED VR

Room Scale àHouse Scale à Warehouse Scale àArena Scale à MR Ubiquity ~5 years

THE VOID

•  Smell of Burnt Marshmallows

•  Misting Effect for Sliming

•  Vibrations in Haptic Vest and Gun

•  Clever Use of Redirected Walking/Elevators

•  Multiplayer

ZERO LATENCY

•  LaserTag-Type Arena Focus

•  System Limited To open warehouse setup

•  PS3 Cameras and Move Controller Tracking

UNIVERSAL ORLANDO HAUNTED HOUSE

•  Similar to Void with tracking

•  Real Actors Involved

•  Mixed Experience – Putting headset on as part of experience

IMAX VR LOS ANGELES

•  Vive + Star VR Experiences

•  Custom Star Trek, DC SuperHero, John Wick experiences

•  Buy tickets to “rides”

END OF 2018 HARDWARE ANTE

BY 2018 VR/AR HMDs will need to have: •  Wireless (Tracking and Power) •  Mobile, not confined to a space •  Scanning in Real-Time & Sharing Scan Data •  Eye + Facial Gesture Tracking •  Predictive Interface •  180+ Degrees FOV + 8K Resolution •  Small Form Factor (Thick Pair of Sunglasses)

THE STANDALONE HMDS OF 2018

HTC, Lenovo, Intel, Samsung, Asus, Qualcomm…

Apple?…Amazon?…Sony?...Avegant?...Magic Leap?

BEYOND THE BASELINE

THANK YOU

Questions?

Chris Wren wwren@gmu.edu chris@wrenar.com chris@notiontheory.com

game.gmu.edu www.wrenar.com

www.notiontheory.com

APPENDIX A

Sports Examples

BASKETBALL

SOCCER

APPENDIX B

Augmented Reality HMDS and USES

AUGMENTED REALITY

Recognition

WINDOWS HOLOGRAPHIC

MS not even talking about VR anymore - only Mixed Reality

INTEL PROJECY ALLOY

Intel tech is similar to Hololens - Real Sense

AR HMDS

Meta 2

AR HMDS

Hololens

AR HMDS

Daqri

AR HMDS

ODG

AR HMDS

Epson

AR HMDS

Atheer

AR HMDS

Occipital Bridge

AR HMDS

Vuzix

APPLE’S SECRET EFFORT

“The next big thing, like the cellphone” - Tim Cook

SAMSUNG AR

Monitorless

AR: FEATURES

●  Recognition ●  Using Real World Data To Drive Graphical Experiences ●  Integrating Graphics into Real World Environments

LYTRO

Capturing & Processing Ultra HD lightfield video

AR: HAND GESTURES

Ability to See and Distinguish Hands and Gestures

AR: LEGO

LEGO

AR: BOOKS & MAGAZINES

AR: SHOPPING

ALIBABA BUY+