“Using eyes, hands and brain for 3D interaction with...
Transcript of “Using eyes, hands and brain for 3D interaction with...
![Page 1: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/1.jpg)
1
1Anatole Lécuyer, HDR defense, June 18th 2010
Habilitation à diriger des recherches
“Using eyes, hands and brain for 3D interaction with virtual environments:
a perception-based approach”
Anatole Lécuyer
Habilitation defense, June 18th 2010, INRIA/IRISA Rennes
![Page 2: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/2.jpg)
2
2Anatole Lécuyer, HDR defense, June 18th 2010
Virtual reality?
Strange and controversial terminology, numerous definitions
Source of fascination, phantasm and fear
Strong and well defined scientific field
Numerous and real applications
« Virtual reality is dreams. » Morton Heilig
I-O Display Systems
NASA
![Page 3: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/3.jpg)
3
3Anatole Lécuyer, HDR defense, June 18th 2010
Virtual Reality (VR)
Definition : “a virtual reality system is an immersive system that provides the user with a feeling of presence (the feeling of "being there" in the virtual world) by means of plausible interactions with a synthetic 3D environment simulated in real-time”.
Virtual reality interfaces• Visual displays : stereoscopic 3D display• Haptic interfaces : force and tactile feedback • Brain-Computer Interfaces : control with brain activity
IMMERSION Graz Tech. Univ.Univ. of Illinois
![Page 4: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/4.jpg)
4
4Anatole Lécuyer, HDR defense, June 18th 2010
VR applications :• Medicine (surgical training, reeducation)• Industry (maintenance operations, data visualization)• Entertainment (video games, theme parks)• Arts and design (interactive sketching, CAD, architecture reviews)
VR challenges
1. Hardware 2. Software 3. Interaction Techniques 4. Perception
Mechanics,
Electronics, etc
Computer
Science
Ergonomics,
Computer-Human Interaction
Neuroscience
psycho/physiology
TRANSVERSALITY
5. VR Applications
Researchactivity
![Page 5: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/5.jpg)
5
5Anatole Lécuyer, HDR defense, June 18th 2010
Research objective
Improve 3D interaction with virtual environmentsMaking full use of available interfaces and sensory modalities:Visual, haptic and brain-computer interfaces>> “using eyes, hands and brain”
Open questions1. Sensory feedback : more immersive2. Interaction technique : more efficient
Perception-based approach• Use knowledge in human perception• For design and evaluation • Collaborations : Univ. Paris 5 (J.M. Burkhardt), Collège de France (A.
Berthoz), Univ. Pierre Mendès-France (E. Gentaz), Freiburg Univ. (J. Wiener), INSERM (O. Bertrand, J.P. Lachaux), etc
![Page 6: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/6.jpg)
6
6Anatole Lécuyer, HDR defense, June 18th 2010
Overview of research activity
1. Novel approaches for visual and haptic feedback of VE• Visual feedback based on user’s gaze• Haptic feedback : Spatialized Haptic Rendering• Combination of visual and haptic feedback : Pseudo-Haptic Feedback
2. Optimal integration of visual and haptic interfaces in VR• Software architecture : multimodal rendering of contacts• Hardware configuration : influence of spatial delocation• Visuo-haptic interaction techniques : Haptic Hybrid Control
3. Toward brain-based interaction with VE• Signal-processing techniques for BCI• Interaction techniques based on BCI• Evaluations of BCI use• Performance models for BCI
![Page 7: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/7.jpg)
7
7Anatole Lécuyer, HDR defense, June 18th 2010
Vision
Virtual environment
Haptic Interface
Visual Interface
Visuo-haptic integrationBrain
Haptic feedback
Interaction TechniquePseudo-haptics
Visual attention
Motor action
Touch
Visual feedback
Mental activity Brain-Computer Interface
(1) Sensory feedback(2) Integration of visual and haptic interfaces
(3) Brain-based interaction
Research framework
![Page 8: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/8.jpg)
8
8Anatole Lécuyer, HDR defense, June 18th 2010
Overview of research activity
1. Novel approaches for visual and haptic feedback of VE• Visual feedback: based on user’s gaze• Haptic feedback : Spatialized Haptic Rendering• Combination of visual and haptic feedback : Pseudo-Haptic Feedback
2. Optimal integration of visual and haptic interfaces in VR• Software architecture : multimodal rendering of contacts• Hardware configuration : influence of spatial delocation• Visuo-haptic interaction techniques : Haptic Hybrid Control
3. Toward brain-based interaction with VE• Signal-processing techniques for BCI• Interaction techniques based on BCI• Evaluations of BCI use• Performance models for BCI >> papers, manuscript
![Page 9: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/9.jpg)
9
9Anatole Lécuyer, HDR defense, June 18th 2010
Overview of research activity
1. Novel approaches for visual and haptic feedback of VE• Visual feedback: based on user’s gaze• Haptic feedback : Spatialized Haptic Rendering• Combination of visual and haptic feedback : Pseudo-Haptic Feedback
2. Optimal integration of visual and haptic interfaces in VR• Software architecture : multimodal rendering of contacts• Hardware configuration : influence of spatial delocation• Visuo-haptic interaction techniques : Haptic Hybrid Control
3. Toward brain-based interaction with VE• Signal-processing techniques for BCI• Interaction techniques based on BCI• Evaluations of BCI use• Performance models for BCI
![Page 10: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/10.jpg)
10
10Anatole Lécuyer, HDR defense, June 18th 2010
Vision
Virtual environment
Haptic Interface
Visual Interface
Visuo-haptic integrationBrain
Haptic feedback
Interaction TechniquePseudo-haptics
Visual attention
Motor action
Touch
Visual feedback
Mental activity Brain-Computer Interface
Visual feedback
![Page 11: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/11.jpg)
11
11Anatole Lécuyer, HDR defense, June 18th 2010
Visual feedback of virtual environments
Computer Graphics• Long history of research [Foley95] [Watt99] [Shirley05]• Impressive results :
synthetic images ~ real images
Interactive 3D graphics and VE>> Real-time, interactivity constraints• Augmented computation capacities [Pharr05]• Interactive visual effects [Guitton95] [Drettakis97]• Perception-based rendering [Mulder00] [Devlin05]
Classical loop • Visual feedback based on user’s actions • >> Head motions, hands actions
![Page 12: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/12.jpg)
12
12Anatole Lécuyer, HDR defense, June 18th 2010
Novel visual feedback loop1. Gaze tracking2. Visual feedback based on perception3. Automatic adaptation of visual feedback to user’s gaze
Visual feedback based on user’s gaze
1
2
3
![Page 13: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/13.jpg)
13
13Anatole Lécuyer, HDR defense, June 18th 2010
Step 1: Gaze-tracking in VE
Existing methods :
Hardware = gaze-trackers>> Numerous systems [Glenstrup95] [Bohme06]• Intrusive vs remote technology• (+) efficient,• (-) expensive, cumbersome
Software = visual attention models>> Numerous models [Lee09] [Itti98] [Longhurst06]• Bottom-up component (color, depth, motion, etc)• Top-down component (memory, habituation,
spatial context, etc)• (+) no hardware,• (-) not real-time, not designed for 3D VE
[Yee01]
[Hillaire08]
![Page 14: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/14.jpg)
14
14Anatole Lécuyer, HDR defense, June 18th 2010
Step 1: Combination of gaze-tracking and visual attention models
Objective : improve gaze-tracking• Combine and associate both technique• Improve overall performance
Method Read gaze tracking output (pixel P) Compute uncertainty window (Wu) Use visual attention model inside Wu Select the most salient pixel (new P)
Evaluation• Comparison with gaze tracker alone• Improve performance of low-cost tracker
(Hillaire et al., Computer Graphics Forum, 2010)
![Page 15: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/15.jpg)
15
15Anatole Lécuyer, HDR defense, June 18th 2010
Objective : simulation of visual perception properties
Effect : Depth-of-Field visual blur
Implementation : • Lens model [Potmesil81] • Auto-focus zone• GGPU technique
Evaluation : • Interactive, real-time effect• Well appreciated (Hillaire et al., ACM VRST, 2007)(Hillaire et al., IEEE CG&A, 2008)
Step 2: Perception-based visual feedback
![Page 16: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/16.jpg)
16
16Anatole Lécuyer, HDR defense, June 18th 2010
Step 3: Automatic adaptation of visual effects to user’s gaze
>> Results: strong subjective preference (Hillaire et al., IEEE VR, 2008)
![Page 17: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/17.jpg)
17
17Anatole Lécuyer, HDR defense, June 18th 2010
Overview of research activity
1. Novel approaches for visual and haptic feedback of VE• Visual feedback based on user’s gaze• Haptic feedback : Spatialized Haptic Rendering• Combination of visual and haptic feedback : Pseudo-Haptic Feedback
2. Optimal integration of visual and haptic interfaces in VR• Software architecture : multimodal rendering of contacts• Hardware configuration : influence of spatial delocation• Visuo-haptic interaction techniques : Haptic Hybrid Control
3. Toward brain-based interaction with VE• Signal-processing techniques for BCI• Interaction techniques based on BCI• Evaluations of BCI use• Performance models for BCI
![Page 18: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/18.jpg)
18
18Anatole Lécuyer, HDR defense, June 18th 2010
Vision
Virtual environment
Haptic Interface
Visual Interface
Visuo-haptic integrationBrain
Haptic feedback
Interaction TechniquePseudo-haptics
Visual attention
Motor action
Touch
Visual feedback
Mental activity Brain-Computer Interface
Combination of visual and haptic feedback
![Page 19: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/19.jpg)
19
19Anatole Lécuyer, HDR defense, June 18th 2010
Pseudo-haptic feedback
Novel approach for haptic (visuo-haptic) feedback
Initial idea (Lécuyer et al., IEEE VR, 2000)• Simulate haptic feedback without a haptic interface• Compatible with passive input devices (mouse)• Use visual feedback to generate haptic illusions
Concept refinement (Lécuyer, Presence, 2009)• ”Pseudo-haptic feedback corresponds to the perception of a haptic
property that differs from the physical environment, by combining visual and haptic information and proposing a new coherent representation of the environment.“
Logitech
![Page 20: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/20.jpg)
20
20Anatole Lécuyer, HDR defense, June 18th 2010
Example: pseudo-haptic textures
Objective: Feel the texture of an image with a mouse
Method• Change cursor’s motion (speed)• C/D gain function of height/depth• Patent (2004)
Experimental evaluation• Ability to perceive bumps/holes• Fine perception
Unchanged motion of the cursor
Decelerated motion
Bump (as displayed on the screen, i.e. in top-view)
Unchanged motion of the cursor
Accelerated motion
(Lécuyer et al., ACM SIGCHI, 2004)
Green cursor
White mask
![Page 21: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/21.jpg)
21
21Anatole Lécuyer, HDR defense, June 18th 2010
History
Foundation = my PhD Thesis (1998-2001)
Students = PhD (Dominjon, Bibin) and Master (Tan)
Collaborations = Univ. Paris 5, UPMF, CLARTE, AFPA, etc
Dissemination = tutorials on « Perception-based haptic rendering » (EUROHAPTICS 2006, IEEE VR 2007, IEEE VR 2008)
Active field = University of British Columbia, Lund University, Fraunhofer, etc
30+ papers, recent survey (Lécuyer, Presence, 2009)
![Page 22: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/22.jpg)
22
22Anatole Lécuyer, HDR defense, June 18th 2010
Overview of research and applications
Studies on haptic properties• Friction (IEEE VR 2000)• Stiffness (IEEE VR 2001)• Mass (IEEE VR 2005)• Textures (ACM CHI 2004) (Eurohaptics 2010)
Studies on applications• SAILOR medical simulator (ACM VRST 2008)• Virtual Technical Trainer « VTT » (IEEE VR 2005) (Eurohaptics 2004)
![Page 23: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/23.jpg)
23
23Anatole Lécuyer, HDR defense, June 18th 2010
Lessons learned
A novel approach for visuo-haptic feedback in VE• Specificity of pseudo-haptic feedback
– Different from : « real » haptics, sensory substitution, tangible interfaces
• Perceptual phenomenon – Illusion or not?– Inter-individual variability– Necessary tuning and calibration
Numerous applications• Pseudo-haptic method?
– Use of input devices (preference for elastic devices)– Use of Control/Display gain
![Page 24: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/24.jpg)
24
24Anatole Lécuyer, HDR defense, June 18th 2010
Overview of research activity
1. Novel approaches for visual and haptic feedback of VE• Visual feedback based on user’s gaze• Haptic feedback : Spatialized Haptic Rendering• Combination of visual and haptic feedback : Pseudo-Haptic Feedback
2. Optimal integration of visual and haptic interfaces in VR• Software architecture : multimodal rendering of contacts• Hardware configuration : influence of spatial delocation• Visuo-haptic interaction techniques : Haptic Hybrid Control
3. Toward brain-based interaction with VE• Signal-processing techniques for BCI• Interaction techniques based on BCI• Evaluations of BCI use• Performance models for BCI
![Page 25: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/25.jpg)
25
25Anatole Lécuyer, HDR defense, June 18th 2010
Vision
Virtual environment
Haptic Interface
Visual Interface
Visuo-haptic integrationBrain
Haptic feedback
Interaction TechniquePseudo-haptics
Visual attention
Motor action
Touch
Visual feedback
Mental activity Brain-Computer Interface
Integration of visual and haptic interfaces
![Page 26: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/26.jpg)
26
26Anatole Lécuyer, HDR defense, June 18th 2010
Combination of visual and haptic interfaces
Spatial problem : Different sizes of haptic and visual workspaces • Haptic workspace is smaller• Example : a PHANToM in a CAVE
Hardware solutions1. Scale 1 haptics [Borro04] [Dominjon07]2. Wearable haptics [Nitzsche03]>> no simple solution
Software solutions1. Scaling techniques [Fischer03]2. Clutching techniques [Johnsen71]>> trade-off = precision / time
>> novel approach : Haptic Hybrid Control
[Fischer03]
Haptic Visual
CEA
![Page 27: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/27.jpg)
27
27Anatole Lécuyer, HDR defense, June 18th 2010
Context: point-based manipulations (translations, 3DOF haptics)
Method• Manipulation workspace = « Bubble »• Hybrid Position/Rate Control• Visual and Haptic display of boundary
The « Bubble » technique
(Dominjon et al., Visual Computer, 2007)(Dominjon et al., Worldhaptics, 2005)
Rate control
Position control
![Page 28: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/28.jpg)
28
28Anatole Lécuyer, HDR defense, June 18th 2010
Bubble technique
Evaluation• Setup : PHANToM in a CAVE• Task : Painting a 3D model• Conditions : clutching, scaling, bubble
Results• Faster, more precise, preferred
(Dominjon et al., LNCS, 2006)
![Page 29: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/29.jpg)
29
29Anatole Lécuyer, HDR defense, June 18th 2010
Haptic Hybrid Rotations
Context : 3D object manipulations (rotations)
Method : DOF separation > cone (2DOF), springs (roll DOF)
Results : fast, precise, preferred (Dominjon et al., IEEE VR, 2006)
![Page 30: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/30.jpg)
30
30Anatole Lécuyer, HDR defense, June 18th 2010
Haptic Hybrid Control
Generic approach for visuo-haptic setup• Hybrid position/rate control• Visual display of the boundary between the two control zones • Haptic display of the boundary with force-feedback
2 Implementations• Bubble (translations)• Haptic Hybrid Rotations (rotations)
Other studies• Tactile pad, 2D Desktop [Casiez07]• Pen, hand-held computers [Hachet08]
Technological transfer • VIRTUOSE API (Haption Company)
(Dominjon et al., Visual Computer, 2007)
![Page 31: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/31.jpg)
31
31Anatole Lécuyer, HDR defense, June 18th 2010
Overview of research activity
1. Novel approaches for visual and haptic feedback of VE• Visual feedback based on user’s gaze• Haptic feedback : Spatialized Haptic Rendering• Combination of visual and haptic feedback : Pseudo-Haptic Feedback
2. Optimal integration of visual and haptic interfaces in VR• Software architecture : multimodal rendering of contacts• Hardware configuration : influence of spatial delocation• Visuo-haptic interaction techniques : Haptic Hybrid Control
3. Toward brain-based interaction with VE• Signal-processing techniques for BCI• Interaction techniques based on BCI• Evaluations of BCI use• Performance models for BCI
![Page 32: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/32.jpg)
32
32Anatole Lécuyer, HDR defense, June 18th 2010
Vision
Virtual environment
Haptic Interface
Visual Interface
Visuo-haptic integrationBrain
Haptic feedback
Interaction TechniquePseudo-haptics
Visual attention
Motor action
Touch
Visual feedback
Mental activity Brain-Computer Interface
Brain-based interaction
![Page 33: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/33.jpg)
33
33Anatole Lécuyer, HDR defense, June 18th 2010
Brain-Computer Interfaces (BCI)• Novel input interface [Vidal73] [Wolpaw02]• Mental tasks : Motor Imagery (MI), attention signals, etc
Limited previous work in BCI and VR • BCI for VR : basic tasks [Bayliss03] [Lalor05]• VR for BCI : learning/motivation [Friedman07]
Problem : small number of (robust) commands
Challenges(1) Neuroscience/electrophysiology studies(2) Peripherals and mental sensors(3) Brain signal processing techniques(4) Novel and more adapted interaction paradigms
Brain-computer interfaces and VR
(Lécuyer et al, IEEE Computer, 2009)
[Friedman07]
[Leeb07]
![Page 34: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/34.jpg)
34
34Anatole Lécuyer, HDR defense, June 18th 2010
BCI-based interaction techniques
Problem : small number of mental commands
Sate-of-the-art in Navigation [Scherer08]• 3 mental states : Left/Right hand, Feet MI• 3 commands : turn left/right, advance
Novel concept : high-level orders• 3 mental states : Left/Right hand, Feet• Selection of target point (binary tree)• Automatic transportation
Evaluation• Task : Navigation• 2 conditions : SoA [Scherer08], High-level
Results • High-level orders faster that soa
(Lotte et al., Presence, 2010)
[Scherer08]
![Page 35: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/35.jpg)
35
35Anatole Lécuyer, HDR defense, June 18th 2010
BCI evaluationProblem : limitations of current evaluations
• Few subjects, lab conditions, intense training
Need for large-scale studies [Guger03]• Assess BCI usage and potential• Real-life conditions (limited learning, out the lab)
Large-scale evaluation (n = 21)• Entertaining application : “use the force”• Lift a virtual spaceship with feet MI
Results• Real ~50 % success • Imagined ~25 % success• Importance of feedback
Cité des sciences (since May)
(Lotte et al., BCI Workshop, 2008)
![Page 36: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/36.jpg)
36
36Anatole Lécuyer, HDR defense, June 18th 2010
OpenViBE software
Integration platform • Open-source software • Design, test and use BCI• Real-time processing of cerebral data
ANR Projects• OpenViBE (2005-2009), OpenViBE2 (2009-2012)
Manpower• 12 men.year, • 3 developers (full-time)
Diffusion• 1st release in May 2009• Numerous users • http://openvibe.inria.fr
(Renard et al., Presence, 2010)
![Page 37: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/37.jpg)
37
37Anatole Lécuyer, HDR defense, June 18th 2010
Perspectives
![Page 38: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/38.jpg)
38
38Anatole Lécuyer, HDR defense, June 18th 2010
Summary of results
Perception-based rendering of virtual environments• Interactive visual feedback based on user’s gaze: (1) novel
techniques for gaze tracking, (2) novel visual effects such as depth-of-field blur
• Novel visuo-haptic technique: “pseudo-haptic feedback” using vision to distort haptic perception
>> Perspectives• Novel visual attention models: more adapted to VE (taking into
account VR interaction), simulating eyes movements (saccades, smooth pursuits, etc). Novel visual effects : realistic camera motions.
• Investigation of pseudo-haptic phenomenon: other modalities (auditory?), perceptual perspective/understanding (integration model? use of other tools: neuroimagery, electromyography, etc)
![Page 39: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/39.jpg)
39
39Anatole Lécuyer, HDR defense, June 18th 2010
Summary of results
Optimal combination of visual and haptic interfaces• Novel interaction paradigm: “Haptic Hybrid Control” to solve
problems related to spatial discrepancies
>> Perspectives• Study of temporal delay : perceptual integration models (Postdoc Nizar
Ouarti, collaboration LPPA/Alain Berthoz)? Design guidelines?• Next generation of visuo-haptic interfaces/hardware
![Page 40: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/40.jpg)
40
40Anatole Lécuyer, HDR defense, June 18th 2010
Summary of results
BCI-based interaction with virtual environments• Novel 3D interaction techniques based on high-level orders to make
up for small number of commands• Large-scale BCI evaluations assessing the usage and potential of BCI
>> Perspectives : • Hybrid BCI : « combining rather than selecting », with multiple signal
processing and multiple brain signals• Advanced sensory feedback : using VR for improved motivation,
closing the loop with multi-sensory rendering
![Page 41: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/41.jpg)
41
41Anatole Lécuyer, HDR defense, June 18th 2010
Technological transfer
Design of VR applications/prototypes • Medical simulator, aeronautics maintenance,
vocational training, assistance to disabled, etc
Patents • (1) pseudo-haptics, (2) haptic motion
Softwares • OpenViBE, SAMIRA (EADS/Airbus),
Multimodal rendering platform (CEA)
Industrial collaborations (PhD/projects) • Orange Labs, CEA, UBISOFT, etc
Transfer of interaction techniques • « Haptic Hybrid Control » (Haption
company), « Magic Barrier Tape » (CEA) (Sreng et al., IEEE TVCG, 2006)(Sreng et al., ACM VRST, 2007)
(Lécuyer et al., IEEE VR, 2003)
![Page 42: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/42.jpg)
42
42Anatole Lécuyer, HDR defense, June 18th 2010
Long-term perspectives
Incorporate other modalities..• Body motion, Feet :
interaction and feedback• « Natural Interactive Walking »
EU Project (2009-2011)• PhD theses: Gabriel Cirio,
Léo Terziman
Towards 3D interaction based on perception and cognition..• Implicit BCI : automatic
adaptation of interaction• OpenViBE2
ANR Project (2009-2012)• PhD thesis Laurent George
![Page 43: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/43.jpg)
43
43Anatole Lécuyer, HDR defense, June 18th 2010
Acknowledgements (local)
Colleagues : Bruno ARNALDI, Alain CHAUFFAUT, Rémi COZOT, Stéphane DONIKIAN, Georges DUMONT, Thierry DUVAL, Fabrice LAMARCHE, Maud MARCHAL, Julien PETTRE,
Postdoc : Marco CONGEDO, Zhan GAO, Nizar OUARTI, Tony REGGIA-CORTE, Mingjun ZHONG,
PhD Students : Gabriel CIRIO, Lionel DOMINJON, Laurent GEORGE, Sébastien HILLAIRE, Fabien LOTTE, Jean SRENG, Léo TERZIMAN,
Expert engineers : Lazar BIBIN, Laurent BONNET, Vincent DELANNOY, Yann JEHANNEUF, Yann RENARD, Aurélien VAN LANGHENHOVE,
Master students : Cédric ARROUET, Laurent ETIENNE, Thomas ERNEST, Jean-Marie HENAFF, Ludovic HOYET, Olivier JOLY, Jildaz LEBILLER, Tristan LE BOUFFANT, Jozef LEGENY, Taegi LIM, Bruno RENIER, Jean-Baptiste SAUVAN, Julien SUPPO, Chee-Hian TAN, Sebastien THOMAS, Maxime VIGNON,
![Page 44: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/44.jpg)
44
44Anatole Lécuyer, HDR defense, June 18th 2010
Vision
Virtual environment
Haptic Interface
Visual Interface
Visuo-haptic integrationBrain
Haptic feedback
Interaction TechniquePseudo-haptics
Visual attention
Motor action
Touch
Visual feedback
Mental activity Brain-Computer Interface
(1) Sensory feedback(2) Integration of visual and haptic interfaces
(3) Brain-based interaction
Lionel Dominjon (2004-2007)
Sébastien Hillaire(2007-)
Fabien Lotte (2005-2008)
Jean Sreng(2005-2008)
Acknowledgements
![Page 45: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/45.jpg)
45
45Anatole Lécuyer, HDR defense, June 18th 2010
Thank you
Questions?
![Page 46: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/46.jpg)
46
46Anatole Lécuyer, HDR defense, June 18th 2010
Habilitation à diriger des recherches
“Using eyes, hands and brain for 3D interaction with virtual environments:
a perception-based approach”
Anatole Lécuyer
Habilitation defense, June 18th 2010, INRIA/IRISA Rennes
![Page 47: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/47.jpg)
47
47Anatole Lécuyer, HDR defense, June 18th 2010
Novel visual attention models
Experimental approach : analysis of gaze behavior when navigating in VE• Navigation task : corridor with turns • Gaze recording (gaze-tracker)• Gaze analysis and modelling
Analysis results • Gaze behavior similar in VE: Anticipation of turns [Grasso98],
saccades [Berthoz00], etc
Novel visual attention model• Based on angular velocity • High performance
(Hillaire et al., ACM VRST, 2009)
![Page 48: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/48.jpg)
48
48Anatole Lécuyer, HDR defense, June 18th 2010
Perception-based visual feedback
Objective : Simulation of visual flow and ocular reflexes during walk
Model : Oscillating camera motions
Innovation : Compensation for constant focalization
Results : • Interactive, real-time effect• Subjective preference for sensation of walking• Improvement of perception of traveled distance
(Lécuyer et al., IEEE VR, 2006)(Terziman et al., IEEE VR, 2009)
![Page 49: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/49.jpg)
49
49Anatole Lécuyer, HDR defense, June 18th 2010
Visual feedback based on gaze
Objective: automatic adaptation of visual effects to user’s gaze
Method Extraction of user’s gaze Filtering Modification of visual effects using gaze data:
– Adaptation of camera motions– Adaptation of depth-of-field blur
Results• Subjective preference (Hillaire et al., IEEE VR, 2008)
![Page 50: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/50.jpg)
50
50Anatole Lécuyer, HDR defense, June 18th 2010
Pseudo-haptic feedback
Assertions1. Pseudo-haptic feedback implies one or more sensory conflicts between visual
and haptic information.2. Pseudo-haptic feedback relies on sensory dominance of vision over touch for
spatial properties.3. Pseudo-haptic feedback corresponds to a new, and coherent, representation of
the environment resulting from a combination of haptic and visual information.4. Pseudo-haptic feedback may create haptic illusions, i.e., the perception of a
haptic property different from the one present in the real environment.
![Page 51: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/51.jpg)
51
51Anatole Lécuyer, HDR defense, June 18th 2010
Pseudo-haptic textures: follow-up
Second technique : the “size technique”
Concept : change size of cursor (zoom-in and out)
Evaluation• 5 experiments• Comparison with speed effect• Combination and Conflict
Results• Size dominates speed• Combination performs better
Hole
displayed on the screen (i.e. in top-view)
Disk-shape cursor
Video Bump
Video Conflict (Lécuyer et al., ACM TAP, 2008)
![Page 52: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/52.jpg)
52
52Anatole Lécuyer, HDR defense, June 18th 2010
Visuo-haptic set-up• Widely spread• Applications
Hard challenges• Software issues : common models, global architecture, interoperable or
abstract components.• Hardware issues : spatial and temporal discrepancies between
workspaces, joint usages
Research activity• Software architecture
for visuo-haptic simulations• Integration of visual and haptic
hardware in VR systems
Combination of visual and haptic interfaces
![Page 53: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/53.jpg)
53
53Anatole Lécuyer, HDR defense, June 18th 2010
Haptic Hybrid Rotations
Context : 6DOF Manipulation of objects (rotations)
Method• Hybrid position/rate control• DOF separations • Visual display = cone (2DOF) and springs (roll DOF)• Haptic display = Force-Feedback • Emulation of elastic input device
(Dominjon et al., IEEE VR, 2006)
![Page 54: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/54.jpg)
54
54Anatole Lécuyer, HDR defense, June 18th 2010
Haptic Hybrid Rotations
Evaluation• Setup : Virtuose and monitor• Task : Building pyramid of cubes• Conditions : scaling, clutching, HHR
Results• Faster, more precise, preffered
0
50
100
150
200
250
300
scaling clutching hybrid
total task completion time (s)
(Dominjon et al., IEEE VR, 2006)
![Page 55: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/55.jpg)
55
55Anatole Lécuyer, HDR defense, June 18th 2010
Multimodal rendering of contact
Generic software architecture for multimodal rendering of contact
Examples of multisensory cues• Visual feedback of contacts : glyphs and lights• Auditory feedback of contacts : spatialized sound
(Sreng et al., IEEE TVCG, 2006) (Sreng et al., ACM VRST, 2007)
![Page 56: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/56.jpg)
56
56Anatole Lécuyer, HDR defense, June 18th 2010
Haptic Rendering of Contact Position Using Vibrations
Context : 6DOF Haptic rendering of complex industrial simulations
Objectives• >> Improve perception of impacts in VE• >> Provide contact ‘position information’
Vibration patterns • 6DOF high-frequency force transient• Example : beam model
(Sreng et al., EuroHaptics, 2008)
Force
Time
![Page 57: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/57.jpg)
57
57Anatole Lécuyer, HDR defense, June 18th 2010
Examples of vibration patternsAm
Fr
AmFr
AmCFr
![Page 58: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/58.jpg)
58
58Anatole Lécuyer, HDR defense, June 18th 2010
6DOF extension of concept
« Virtual beam metaphor »
Experimental approach
Promising technique**
Spatialized Haptic Rendering(Sreng et al., IEEE VR, 2009)
![Page 59: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/59.jpg)
59
59Anatole Lécuyer, HDR defense, June 18th 2010
Optimal integration of visual and haptic interfaces
Visuo-Haptic setups:
Strong differences• Temporal delay• Spatial mismatch
Challenges• Perceptual Studies• Novel interfaces and
techniques
Research activitySpatial discrepencies:1. Translational offset
(Congedo et al., Presence, 2006)2. Size difference
![Page 60: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/60.jpg)
60
60Anatole Lécuyer, HDR defense, June 18th 2010
Delocation of visual and haptic interfaces
Delocation: spatial offset between vision and touch
Experimental approach• Perceptual study• Psychophysical experiment• Two conditions : collocated vs delocated• Computation of sensory weights
Results • Weight of vision increases
with delocation
Design guidelines• Strong impact on VR setups• Collocation to promote haptics(Congedo et al., Presence, 2006)
![Page 61: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/61.jpg)
61
61Anatole Lécuyer, HDR defense, June 18th 2010
Improving EEG signal processing
Measurement of brain activity
1 Sensor : ElectroEncephaloGraphy (EEG)
0 500 1000 1500 2000 250050
0
50
C3
0 500 1000 1500 2000 250050
0
50
C4
0 500 1000 1500 2000 250050
0
50
P3
0 500 1000 1500 2000 250020
0
20
P4
0 500 1000 1500 2000 250020
0
20
O1
0 500 1000 1500 2000 250020
0
20
Sample (250 per second)
O2
Brain measures (EEG)
![Page 62: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/62.jpg)
62
62Anatole Lécuyer, HDR defense, June 18th 2010
Improving EEG signal processing
Measurement of brain activity
Preprocessing
Feature extraction
1
2
3
Novel approach using inverse solutions
(Lotte et al., IEEE TNSRE, 2007)(Lotte et al., PMB, 2006)
• Temporal information [Schlogl97] [Penny99]• Frequency information [Pfurtscheller01] [Palanappian05]
![Page 63: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/63.jpg)
63
63Anatole Lécuyer, HDR defense, June 18th 2010
Improving EEG signal processing
Measurement of brain activity
Preprocessing
Classification
Feature extraction
1
2
3
4Results :
• Better performances• Novel capacities : physical meaning, readability, rejection, etc
Linear classifiers, neural networks, nearest neighbor, probabilistic classifiers [Lotte07] [Bashashati07 ]
(Lotte et al., J. Neur. Eng., 2007)
(Lotte et al., IEEE TNSRE, 2007)
(Zhong et al., PRL, 2008)
(Lotte et al., ICPR, 2008)
Survey of existing approaches
Testing novel approaches• Fuzzy Inference Systems :
• Gaussian Process :
• Reject Options :
![Page 64: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/64.jpg)
64
64Anatole Lécuyer, HDR defense, June 18th 2010
Iterative research approach:
Design and evaluation
Research activity1. Interaction techniques
– Small number of commands
2. Evaluations– Performance,
preference, usages3. Performance models
– Predict performance, improve design
BCI-based interaction with VE
![Page 65: “Using eyes, hands and brain for 3D interaction with ...videos.rennes.inria.fr/hdr/lecuyer/Lecuyer_HDR.pdf · Definition : “a virtual reality system is an immersive system that](https://reader033.fdocuments.in/reader033/viewer/2022042320/5f0a873d7e708231d42c13d0/html5/thumbnails/65.jpg)
65
65Anatole Lécuyer, HDR defense, June 18th 2010
Step 3: BCI Performance models
Objective : predict performance and improve design of techniques
“P300-signal” applications• Visual attention on flashing objects (letters, icons, virtual objects) • Object selection triggers a state change [Rebsamen07]
Performance model based on Markov Chains• Time to select a target• Number of flashes needed
Experimental validation• Three different techniques:• Good results: matching
between model and experimental data
(Sauvan et al., ACM CHI, 2009)