Grasping the Future: Virtual Hands Control for Fine Motor Tasks
-
Upload
ronald-punako-jr -
Category
Education
-
view
331 -
download
2
Transcript of Grasping the Future: Virtual Hands Control for Fine Motor Tasks
![Page 1: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/1.jpg)
@IITSEC NTSAToday
Grasping the Future: Virtual Hands Control for Fine Motor Tasks
Ronald Punako Jr., Concurrent Technologies Corporation
![Page 2: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/2.jpg)
@IITSEC NTSAToday
Physical Equipment Training can be Effective, but Cost Prohibitive
2
![Page 3: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/3.jpg)
@IITSEC NTSAToday
How can we Provide Effective Equipment Training at Reduced
Cost?
3
![Page 4: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/4.jpg)
@IITSEC NTSAToday
Use the Same Hands-on Procedures with Virtual Equipment
4
![Page 5: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/5.jpg)
@IITSEC NTSAToday
Overview
Science & Design System Architecture Interactivity Experimental Design Results Future Research
5
![Page 6: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/6.jpg)
@IITSEC NTSAToday
Science & Design – Situated Cognition
Prototype is a virtual reality system that provides both cognitive and psychomotor equipment maintenance training experiences.
Knowing is inseparable from doing. All knowledge is situated in activity bound to social, cultural and physical contexts (Situated Cognition, Gee et al.).
6
![Page 7: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/7.jpg)
@IITSEC NTSAToday
Science and Design – Symbolic Interaction
7
Gestures may be assigned to different actions in a computer based instruction to embody varied content knowledge and to regulate the instruction and learning process (Xu, X., & Ke, F., 2014).
Experience and cognition are grounded in actions and movements, such as simulations, situated actions, or states of body gestures (Barsalou, 2008, 2010).
![Page 8: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/8.jpg)
@IITSEC NTSAToday
Science and Design - Mental Practice
8
When thinking about a saw, the actions and movements using a saw to cut the wood or metal will encode one’s perception on the functionality of a saw.
Body movements and actions also prompt us to recall our prior experiences and cognition, link them to the current situation, and reapply them for future actions and understanding (Barsalou 2010; Mizelle & Wheaton 2010).
![Page 9: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/9.jpg)
@IITSEC NTSAToday
System Architecture – Prototype Client
9
User Interface
Window Manager
CalloutManager
ToolsManager
Scene Graph
EquipmentModel
Parts
Fasteners
Sequencing
Exercise
Sequences
ActionsX
Gaze
Gesture
Talk
![Page 10: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/10.jpg)
@IITSEC NTSAToday
System Architecture - Gestures
10
Human perception of movementFluid motion: 13-50msFrames p/second: 24-60fps
Prototype hand movementFluid motion: 13msFrames p/second: 78fps
![Page 11: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/11.jpg)
@IITSEC NTSAToday
System Architecture - Visualization
11
Helps attain high measures of presence, or the feeling of “being there.”
![Page 12: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/12.jpg)
@IITSEC NTSAToday
System Architecture - Game Engine
12
Unity is the game engine and integration point for the prototype.
![Page 13: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/13.jpg)
@IITSEC NTSAToday
Interactivity - Virtual Environment
13
![Page 14: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/14.jpg)
@IITSEC NTSAToday
Interactivity – Equipment Model
14
![Page 15: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/15.jpg)
@IITSEC NTSAToday
Interactivity - Hands Articulation
15
![Page 16: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/16.jpg)
@IITSEC NTSAToday
Interactivity - Pointing
16
![Page 17: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/17.jpg)
@IITSEC NTSAToday
Interactivity - Pinching
17
![Page 18: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/18.jpg)
@IITSEC NTSAToday
Interactivity - Exercise Progression
18
![Page 19: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/19.jpg)
@IITSEC NTSAToday
Benefits – High-Order Cognitive Skills
19
Application of high-order cognitive activities Apply, Analyze, and Evaluate
Immediate application of procedural knowledge Consistent and constant feedback
during procedure
Prototype
![Page 20: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/20.jpg)
@IITSEC NTSAToday
Benefits – Low-Order Cognitive Skills
20
Application of low-order cognition activities Remember, understand
Why? It’s a virtual instruction manual 3D collapsed\exploded diagram Parts name callouts using gaze
Prototype
![Page 21: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/21.jpg)
@IITSEC NTSAToday
Benefits – Psychomotor Skills
21
Imitation of tool activity improves skilled movements Psychomotor Skills (by rehearsing motions) Visuospatial Skills (by rehearsing motions in
realistic environment) Visual correlation of real-to-virtual hand
movement creates the illusion of “being there”
Hand movement and head movementcreates the illusion the body in VR space
Prototype
![Page 22: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/22.jpg)
@IITSEC NTSAToday
Experimental Design - Environment
22
OculusLeap MotionAudio
Starting location
Head tracker
Alienware PC5 Feet
1in. 2ft.
Gesture frame
![Page 23: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/23.jpg)
@IITSEC NTSAToday
Experimental Design – Sample Frame
23
Participants were highly familiar and experienced with the helicopter mission kit (HMK).
Key: pn = participant code
![Page 24: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/24.jpg)
@IITSEC NTSAToday
Experimental Design – Immersive Tendencies Questionnaire (ITQ)
24
Participants reported low to moderately-low immersive tendencies.
Key: pn = participant code Scale 1 = low tendency, 7 = high tendency
![Page 25: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/25.jpg)
@IITSEC NTSAToday
Experimental Design – Light Bulb Task
Start Locate part 1
Remove 7 fasteners
Decouple part 1
25
Locate part 2
Remove 2 fasteners
Decouple part 2
![Page 26: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/26.jpg)
@IITSEC NTSAToday
Results – Presence Questionnaire (PQ)
26
Participants reported high to moderately-high levels of presence.Recall that ITQ scores are highly correlated to PQ scores.
Key: 1 = Scale reversed
![Page 27: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/27.jpg)
@IITSEC NTSAToday
Results - Thinking Out Loud
27
Key: tn = Time scale
Mostly positive experiences reported for Realism.Some positive and negative experiences reported for Interface Quality.
![Page 28: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/28.jpg)
@IITSEC NTSAToday
Results – Presence Discussion
Low ITQ outcomes are highly correlated to low PQ outcomes, and vice-versa. However, participants reported low ITQ outcomes and high PQ outcomes.
1. Would the same participants report a low PQ outcome without virtual hands?
2. Would a larger and more diverse participant sample report differently?
28
![Page 29: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/29.jpg)
@IITSEC NTSAToday
Results – Skills Discussion
Regarding the efficacy of cognitive and psychomotor skills training.
1. Does use of virtual hands training provide better transfer of cognitive skills than non-virtual hands implementations?
2. Does use of virtual hands provide better transfer of psychomotor skills than mental practice?
29
![Page 30: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/30.jpg)
@IITSEC NTSAToday
Future Research
1. How does tool simulation effect cognitive and psychomotor tasks?
2. How does eye movement support measurement of mastery?
3. How can exercise construction be automated?
4. How may collaboration be supported across realities?
30
![Page 31: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/31.jpg)
@IITSEC NTSAToday
Questions
Thank you for your time!
31
![Page 32: Grasping the Future: Virtual Hands Control for Fine Motor Tasks](https://reader036.fdocuments.in/reader036/viewer/2022062223/58721cee1a28ab3f188b784d/html5/thumbnails/32.jpg)
@IITSEC NTSAToday
Contacts
Corey NorrisDirector, Software [email protected]
Ron PunakoSenior Software [email protected]
32