Immersive 3D Environment Using Kinect and Voice Commands
-
Upload
kinda-altarbouch -
Category
Education
-
view
93 -
download
1
Transcript of Immersive 3D Environment Using Kinect and Voice Commands
![Page 1: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/1.jpg)
Utilizing Kinect control for a more immersive interaction
with 3D environments
Supervised byDr. Ammar Joukhadar Dr. Noor Shaker Eng. Mohammad Shaker
Designed, Implemented and Tested byMohammed Saed Haj Ali Marah Halawa Kinda Tarboush
F.I.T.E of Damascus, Syria – AI Department 2014
![Page 2: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/2.jpg)
Content
• Motivation
• The Problem with Today’s Games
• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through
in-game Interactions• Hand Gesture • Voice Command
• Our Application
• Future Perspectives
• Demo
![Page 3: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/3.jpg)
Content
• Motivation
• The Problem with Today’s Games
• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through
in-game Interactions• Hand Gesture • Voice Command
• Our Application
• Future Perspectives
• Demo
![Page 4: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/4.jpg)
“Why using astylus when Godgave us tenfingers!” Steve Jobs when
introducingthe first iPhone, 2010
![Page 5: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/5.jpg)
![Page 6: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/6.jpg)
Good design looks great, yes,
![Page 7: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/7.jpg)
Good design looks great, yes, but
![Page 8: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/8.jpg)
smell great and sound great?
Good design looks great, yes, but
why shouldn't it also feel great,
![Page 9: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/9.jpg)
smell great and sound great?
Good design looks great, yes, but
why shouldn't it also feel great,
![Page 10: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/10.jpg)
Jinsop Lee
“Design for all 5 senses”
The Concept of
![Page 11: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/11.jpg)
Content
• Motivation
• The Problem with Today’s Games
• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through
in-game Interactions• Hand Gesture • Voice Command
• Our Application
• Future Perspectives
• Demo
![Page 12: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/12.jpg)
The player mostly has only one method to interact with the game (most games has the
same playing peripherals) like joystick or mouse
![Page 13: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/13.jpg)
Most games don’t let the player to create their own content\ worlds and share it with others.
![Page 14: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/14.jpg)
For most games there’s no direct interaction between the game and the player (only one way
from the player to the game)
![Page 15: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/15.jpg)
For most games there’s no direct interaction between the game and the player (only one way
from the player to the game)
![Page 16: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/16.jpg)
A Game For You to Use Your
Senses
![Page 17: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/17.jpg)
Content
• Motivation
• The Problem with Today’s Games
• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through
in-game Interactions• Hand Gesture • Voice Command
• Our Application
• Future Perspectives
• Demo
![Page 18: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/18.jpg)
Sketching a Level
Sketch Your Level
![Page 19: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/19.jpg)
Using Kinect
![Page 20: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/20.jpg)
Interaction with the environment
Hand Gesture
![Page 21: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/21.jpg)
Interaction with the environment
Voice Command
![Page 22: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/22.jpg)
Our Game
Put here a picture or a short video of the game. Just so that people know what are you talking about
![Page 23: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/23.jpg)
The Immersion Process
Player Sketch
Generating basic Level
Canvas
Allow Edit and
Generation of Content
![Page 24: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/24.jpg)
The Immersion Process
Player Sketch
Generating basic Level
Canvas
Allow Edit and
Generation of Content
![Page 25: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/25.jpg)
The Immersion Process
Player Sketch
Generating basic Level
Canvas
Allow Edit and
Generation of Content
![Page 26: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/26.jpg)
Sketching a Level
Put here a picture of a sketch and a picture of the height map of the same sketch
![Page 27: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/27.jpg)
The Immersion Process
Player Sketch
Generating basic Level
Canvas
Allow Edit and
Generation of Content
![Page 28: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/28.jpg)
Sketch to Level Conversion
Talk about the conversion from sketch to level and the cuboid algorithm you did (with picture)
![Page 29: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/29.jpg)
The Immersion Process
Player Sketch
Generating basic Level
Canvas
Allow Edit and
Generation of Content
![Page 30: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/30.jpg)
Hand Gesture
Interaction with
Dynamic Hand
Gesture
Interaction with
Static Hand
Gesture
![Page 31: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/31.jpg)
Dynamic Hand Gesture
• Animated gestures
![Page 32: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/32.jpg)
Static Hand Gesture
• Comparison with reference images• Using machine learning• Rule based system
![Page 33: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/33.jpg)
SelectRotateDelete Move
The gestures that we will
use
![Page 34: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/34.jpg)
Hand gesture in our application
Dynamic Hand
Gesture
Static Hand
Gesture
![Page 35: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/35.jpg)
Hand gesture in our application
Dynamic Hand Gesture
Discrete Markov
Processes
Static Hand Gesture
Rule based system
![Page 36: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/36.jpg)
Hand gesture in our application
![Page 37: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/37.jpg)
Interaction via Voice Commands
![Page 38: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/38.jpg)
Voice Command
• Audio to text conversion
![Page 39: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/39.jpg)
Voice Command
• Speech Recognition Hidden Markov Model
![Page 40: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/40.jpg)
Voice Command
• Words Meaning Conceptual Grammar
![Page 41: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/41.jpg)
Content
• Motivation
• The Problem with Today’s Games
• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through
in-game Interactions• Hand Gesture • Voice Command
• Our Application
• Future Perspectives
• Demo
![Page 42: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/42.jpg)
Our Application
Generate 3D environment and edit it using most of your senses!
![Page 43: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/43.jpg)
Content
• Motivation
• The Problem with Today’s Games
• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through
in-game Interactions• Hand Gesture • Voice Command
• Our Application
• Future Perspectives
• Demo
![Page 44: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/44.jpg)
Future Perspective
• Enhance the social aspect of the game• Multiple player playing at the same time at
the same level
• Assess the game immersions metrics
• Allow for more immersive and enhanced gestures
• Analyze the applicability of the approach on other hardware (like Oculus Rift)
• Analyze the applicability of the approach on other game genres
![Page 45: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/45.jpg)
Content
• Motivation
• The Problem with Today’s Games
• Modeling• Sketching a Level• Generating the Basic Level Structure• Allow Edit and Creation on Content through
in-game Interactions• Hand Gesture • Voice Command
• Our Application
• Future Perspectives
• Demo
![Page 46: Immersive 3D Environment Using Kinect and Voice Commands](https://reader031.fdocuments.in/reader031/viewer/2022032623/55d0699ebb61eb337e8b46af/html5/thumbnails/46.jpg)
Thanx for listening!