Binocular Eye Trackingand Calibration in Head-mounted Displays

50
An Affordable Solution for Binocular Eye Tracking and Calibration in Head-mounted Displays Michael Stengel (TU Braunschweig), Steve Grogorick (TU Braunschweig), Elmar Eisemann (TU Delft), Martin Eisemann (TH Köln), Marcus Magnor (TU Braunschweig) ACM Multimedia 2015, Brisbane Michael Stengel stengel @ cs.tu-bs.de

Transcript of Binocular Eye Trackingand Calibration in Head-mounted Displays

An Affordable Solution for Binocular Eye Tracking and Calibration in Head-mounted Displays

An Affordable Solution for Binocular Eye Trackingand Calibration in Head-mounted DisplaysMichael Stengel (TU Braunschweig), Steve Grogorick (TU Braunschweig),Elmar Eisemann (TU Delft), Martin Eisemann (TH Kln), Marcus Magnor (TU Braunschweig)ACM Multimedia 2015, Brisbane

Michael Stengelstengel @ cs.tu-bs.de

1

Immersive media is getting popular.Photographic footage by shutterstock.com

2

Unreal Engine and footage. Copyright by Epic Games, International.Support for Real-time gaze-tracking is missing !

3

binocular eye-trackinginexpensive components< 400 US$low latency ( 12-17 ms)modular designefficient user calibrationfull field of view (100 per eye)uniform tracking accuracyfully adjustable lensesno external camerasOur solution.

4

BINOCULAR EYE TRACKING PROTOTYPEHMD TECHNOLOGY - TODAY AND TOMORROWCALIBRATION AND TRACKINGVR APPLICATIONS

5

HMD Technology Today.Wide Field of View

LEFT EYERIGHT EYEFOVEAL REGIONHMD SCREEN> 100 FoV per eye to create immersionbiconvex aspheric lensesfight chromatic abberation in renderinglarge viewing angles require higher FoV

6

HMD Technology Today.Wide Field of ViewHigh temporal resolution

refresh rate at least above retinal integration ratestrobe displays largely reduce display motion bluroptimal display : 1000 HzTemporal Video Filtering for Perceptual Motion Blur[Stengel et al. 14]

7

HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolution

Apparent Display Resolution Enhancement[Didyk11, Templin12, Stengel13]2.5k per eye is state-of-the-artnot sufficient for foveal region (0.6 0.9 arc minutes) requires pixel resolution of 16k x 16kLena Sderberg. Copyright by USC.

8

HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolutionInstant visual feedback

Kite. Copyright by Epic Games, Inc.Foveated Gaze-Contingent Displays[Dorr et al. 06, Duchowski et al. 07, Targino da Costa,14]tolerable latency depends on application< 20 to 40 ms to avoid motion sicknessrequires serious render optimization features

9

HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolutionInstant visual feedbackAccurate physical viewpoint

Low latency head trackingnot sufficiently solved for mobile systemsLarge-Scale Direct Monocular SLAM [Engel et al. 14]

10

HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolutionInstant visual feedbackAccurate physical viewpointNatural depth perception (accomodation)

Pursuit Gaze Calibration [Pfeuffer et al.13]Focus Cues Displays [Narain et al. 15, Huang et al. 15]interpupillary distance estimation (IPD)stereo rendering using IPDnatural accomodation currently not supported (no natural depth of field)

11

HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolutionInstant visual feedbackAccurate physical viewpointNatural depth perception (accomodation)High dynamic range imaging

A computational model of afterimages[Ritschel et al. 2012]Simulating the Visual Experience, Tone-Mapping[Benoit et al. 09 , Ritschel et al. 12, Jacobs et al. 15]Currently global tone mapping

12

HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolutionInstant visual feedbackAccurate physical viewpointNatural depth perception (accomodation)High dynamic range imagingPerceptual realismSmart actors; collaborative VR with gaze support[Marsella et al. 13, Debevec et al. 12, Duchowski et al. 05, Steptoe et al. 08]

Digital Ira Project [Activision, Inc. and USC, 2012]

13

HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolutionInstant visual feedbackAccurate physical viewpointNatural depth perception (accomodation)High dynamic range imagingPerceptual realism

Benefits fromgaze-tracking !

Gaze-tracking is the technologyyou are looking for.

14

ERGONEERSFOVESMI DK1 EYE TRACKING UPGRADEARRINGTON RESEARCH EYE TRACKING KIT>20,000 USD50,000 USD15,000 USDNo details. Not yet available.

15

Construction blueprints will be freely available.

16

BINOCULAR EYE TRACKING PROTOTYPEHMD TECHNOLOGYCALIBRATION AND TRACKINGVR APPLICATIONS

17

18

3D-printed body caseFixation using head strapsOrientation tracker5.6 HD DisplayAspheric lenses

19

3D-printed body caseFixation using head strapsOrientation tracker5.6 HD DisplayAspheric lensesEye-tracking cameras

20

3D-printed body caseFixation using head strapsOrientation tracker5.6 HD DisplayAspheric lensesDichroic mirrorsEye-tracking cameras

21

3D-printed body caseFixation using head strapsOrientation tracker5.6 HD DisplayFlexible Lens-holderswith illumination unitsAspheric lensesDichroic mirrorsEye-tracking cameras

22

3D-printed body caseFixation using head strapsOrientation tracker5.6 HD DisplayFlexible Lens-holderswith illumination unitsAspheric lensesPosition-tracking cameraDichroic mirrorsEye-tracking cameras

23

BICONVEX LENSESSCREENCASEUSER

24

INFRARED ILLUMINATIONEYE TRACKINGCAMERAEYE TRACKINGCAMERADICHROICMIRRORDICHROICMIRROR

25

Infrared LightVisible Light

Dichroic mirror behavior

26

BINOCULAR EYE TRACKING PROTOTYPEHMD TECHNOLOGYCALIBRATION AND TRACKINGVR APPLICATIONS

27

PRE-CALIBRATION / HMD

Reprojection error aftercamera calibrationUsed lenses from DK1IOR estimation andphysical renderingLens shape reconstruction

28

Unkown parametersLens settingsEye locations

29

User specificUnkown parametersLens settingsEye locations

30

CALIBRATION / USER

Average eye assumptionUnknown propertyRelative location

PHYSICAL EYE MODELA photon accurate model of the human eye [Deering05]The Physiology of the Eye [Adler et al., 08]

31

CALIBRATION / LENS LOCATION

SIMULATED MODELCAMERA CAPTURELEFT EYERIGHT EYE

32

CALIBRATION / LENS LOCATION

CAMERA CAPTURELEFT EYERIGHT EYESIMULATED MODEL

33

CALIBRATION / EYE BALL LOCATION

CAMERA CAPTURELEFT EYERIGHT EYESIMULATED MODEL Details in the paper

34

CALIBRATION / USER

Interpupillary Distance (IPD)IPD

35

PUPIL TRACKINGPHYSICAL EYE MODEL

Pupil Size

Pupil Position

SCREEN CAPTURE

36

PUPIL TRACKINGLarge occlusionLess occlusionPupilpositionGaze mapping

37

GAZE ACCURACY

ANGULAR GAZE ERROR DISTRIBUTION0.750.500.000.251.00

38

TRACKING PERFORMANCE / DESKTOPProcessing stepTimingsCamera frame capture11 msOcclusion metric3 msPupil Detection (occluded)2 msPupil Detection (visible)1.5 msGaze computation< 0.1ms~ 17.5 ms (total)

may be skippedfor many users~ 12.5 ms (total)

< 2 ms

< 8 mstwo cores on [email protected] algorithmallows gaze-contingency

39

TRACKING PERFORMANCE / MOBILEProcessing stepTimingsCamera frame capture20 msPupil Detection (visible)13.5 msGaze computation< 0.1ms~ 33.5 ms (total)

RASPBERRY PI2quad core processorone core per eye and camera grabberimplementation for NEON instruction set

40

BINOCULAR EYE TRACKING PROTOTYPEHMD TECHNOLOGYCALIBRATION AND TRACKINGVR APPLICATIONS

41

FOVEATED RENDERINGUnreal Engine and footage.Copyright by Epic Games, International.

42

SIMULATED EFFECTS / ACCOMODATIONUnreal Engine and footage.Copyright by Epic Games, International.

43

SIMULATED EFFECTS / ACCOMODATION

Unreal Engine and footage.Copyright by Epic Games, International.

44

AVATAR ANIMATION / TELE PRESENCE

Unreal Engine and footage.Copyright by Epic Games, International.

45

USER STUDIES OF VIRTUAL EXPERIENCES

46

GAZE ANALYSIS IN PANORAMIC VIDEOS[Loewe and Stengel et al.,ETVIS15]

47

MORE APPLICATIONS!Interaction MetaphorSelection / manipulationHands-free navigationAdaptive Tone MappingAdaptive VisualizationFoveated video/streamingExtended interfacesAugmented VRMobile applications

EVE: Valkyrie

48

Thank you.

Michael Stengelstengel @ cs.tu-bs.degraphics.tu-bs.de/people/stengel

Steve Grogorickgrogorick @ cs.tu-bs.degraphics.tu-bs.de/people/grogorick

Elmar Eisemann (TU Delft)Martin Eisemann (TH Kln)Marcus Magnor (TU Braunschweig)

49

REFERENCESApparent Display Resolution Enhancement for Moving Images [Didyk11]Apparent Resolution Enhancement for Animations [Templin12]Optimizing Apparent Display Resolution Enhancement for Arbitrary Videos [Stengel13]A computational model of afterimages [Ritschel et al. 12]Simulating the Visual Experience of Very Bright and Very Dark Scenes [Jacobs et al. 15]Spatio-temporal tone mapping operator based on a retina model [Benoit et al. 09]Foveated Gaze-Contingent Displays for Peripheral LOD Management,3D Visualization, and Stereo Imaging [Duchowski et al. 07]A Retina-Based Perceptually Lossless Limit and a Gaussian FoveationScheme With Loss Control [Targino da Costa,14]Gaze-Contingent Spatio-Temporal Filtering in a Head-Mounted Display [Dorr et al. 06]Pursuit Calibration: Making Gaze Calibration Less Tedious and More Flexible [Pfeuffer et al. 13]Temporal Video Filtering and Exposure Control for Perceptual Motion Blur [Stengel et al. 14]SmartBody, Cerebella Project [Marsella et al., USC]LSD-SLAM: Large-Scale Direct Monocular SLAM [Engel et al., 14]

50