Kinect virtual-learning (SIM U KIN Graduation Project)
-
Upload
mohamed-hesham -
Category
Software
-
view
183 -
download
0
Transcript of Kinect virtual-learning (SIM U KIN Graduation Project)
![Page 1: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/1.jpg)
Kinect Virtual Learning
Team members: Mohamed Hesham Mahmoud Soliman Ahmed Amr Yousef Ahmed Ahmed Nasser Date: 5/7/2015
![Page 2: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/2.jpg)
Agenda• Problem & Solution• Presentation Application • Interviewing Application • Pattern Recognition techniques• Speech Recognition • Demo• Questions ?
![Page 3: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/3.jpg)
Problem• Why Soft Skills ?• Business & Education• Chance !!• Software !!
![Page 4: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/4.jpg)
Existing Solutions• Expensive Courses
• Online Materials
![Page 5: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/5.jpg)
Idea & Solution• Simulate the presentation & interview Processes • train the presenter & interviewee on the correct behavior
Body movements Speech recognition Facial Expression
• Feedback of weakness points
![Page 6: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/6.jpg)
What is Kinect !
![Page 7: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/7.jpg)
Message Delivery • Body Language• Speech • Facial Expression
![Page 8: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/8.jpg)
Project Overview
![Page 9: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/9.jpg)
Presentation Application
![Page 10: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/10.jpg)
Detected Body MistakesReferences
![Page 11: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/11.jpg)
Body Joints
![Page 12: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/12.jpg)
Mistakes Categories
![Page 13: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/13.jpg)
Dynamic Mistakes
![Page 14: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/14.jpg)
Body Language
Body Language
![Page 15: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/15.jpg)
Body Language
Body Language
![Page 16: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/16.jpg)
Body Language
Body Language
![Page 17: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/17.jpg)
Static Mistakes
![Page 18: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/18.jpg)
Body Language
Body Language
![Page 19: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/19.jpg)
Body Language Hand in Pocket
![Page 20: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/20.jpg)
Leaning Body Language
![Page 21: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/21.jpg)
Body Language Hand on waist
![Page 22: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/22.jpg)
Body LanguageUp leg
![Page 23: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/23.jpg)
Presentation Application Flow
![Page 24: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/24.jpg)
![Page 25: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/25.jpg)
![Page 26: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/26.jpg)
![Page 27: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/27.jpg)
![Page 28: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/28.jpg)
![Page 29: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/29.jpg)
![Page 30: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/30.jpg)
![Page 31: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/31.jpg)
![Page 32: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/32.jpg)
![Page 33: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/33.jpg)
Interview Application
![Page 34: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/34.jpg)
References
Detected Face Mistakes
![Page 35: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/35.jpg)
Facial Techniques Face Joints Face Orientation
![Page 36: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/36.jpg)
Face Gestures Touching face and Yawning
![Page 37: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/37.jpg)
Facial Techniques Face Joints Face Orientation
![Page 38: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/38.jpg)
Face Gestures No Eye Contact
![Page 39: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/39.jpg)
Face Gestures
Feeling Shy
![Page 40: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/40.jpg)
Face Gestures Smiling
![Page 41: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/41.jpg)
Face Gestures Cleaning Glasses
![Page 42: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/42.jpg)
Face Gestures Aggressive
![Page 43: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/43.jpg)
Face Gestures Bad Posture
![Page 44: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/44.jpg)
Interview Application Flow
![Page 45: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/45.jpg)
![Page 46: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/46.jpg)
![Page 47: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/47.jpg)
![Page 48: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/48.jpg)
![Page 49: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/49.jpg)
![Page 50: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/50.jpg)
![Page 51: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/51.jpg)
![Page 52: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/52.jpg)
![Page 53: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/53.jpg)
![Page 54: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/54.jpg)
![Page 55: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/55.jpg)
Pattern Recognition Techniques
![Page 56: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/56.jpg)
Classifiers Categories
![Page 57: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/57.jpg)
Rule-Based Classifiers • Conditions
Threshold
• 12 Joints with XYZ Coordinates
![Page 58: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/58.jpg)
Hidden Markov Model Classifiers
• Dataset 50 sample for each mistake
• Models 9 states 200 Observations
• Phases Learning Decoding Evaluate
![Page 59: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/59.jpg)
Rule-Based HMM80
82
84
86
88
90
92
94
96
ArmLegBody center
Classifier Arm Leg Center Type
Rule-Based 92% 88% 90% Static
HMM 95% 85% 93% Static – Dynamic
Results and Statistics
![Page 60: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/60.jpg)
Classifiers Conclusion
• HMM is more accurate than Rule-based and support Dynamic states
• Rule-based is complex to detect specific threshold for different bodies
![Page 61: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/61.jpg)
Gesture Detection Flow
• Not all features are relevant to all gestures.
Each gestures has its own feature vector.
• More than one Gesture can happen at the same time. We group related gestures together under respective limbs.
• Gestures can be related to more than one limb We divide gestures into parts called “states”.
Problems:
![Page 62: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/62.jpg)
Gesture Detection Flow
Right
Arm
Hand Over Hand
Hand On Waist
Hand In PocketLeft Arm
Hand Over Hand
Hand On Waist
Hand In Pocket
Right Leg
Cross Leg
Up Leg
Left Leg
Cross Leg
Up Leg
body
Leaning left
Leaning right
![Page 63: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/63.jpg)
Gesture Detection Flow
Flow :
• Each Gesture has a certain condition on the detected states.• Body consists of 5 Limbs.• Each Limb has a most probable state.• Each state has a classifier object that receives the feature vector.• Each state has its own feature vector.
![Page 64: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/64.jpg)
Gesture Detection Flow
Gesture types :
• Static Gestures : No movement involved Happens when any state from a group of states happen. Happens when all states from a group of states happen.
• Dynamic Gestures : Requires the body to move Happens when a sequence of states happen in a short period of time.
![Page 65: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/65.jpg)
Speech Recognition
![Page 66: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/66.jpg)
Speech Recognition• Speech recognition process
![Page 67: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/67.jpg)
Speech Recognition• Presentation application (Filler Words)
Fillers words and phrases people use to cover verbal gaps—are word crutches. Presenters often use them out of fear.
• The most common fillers are: So, And, All right, Okay, Like, Now, Well, You know,
Right, Um and Uh.
![Page 68: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/68.jpg)
English Test phases
![Page 69: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/69.jpg)
First Phase "Put Question"
![Page 70: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/70.jpg)
Second Phase "Paragraph Test "
![Page 71: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/71.jpg)
Third Phase “Knowledge "
![Page 72: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/72.jpg)
Conclusion• HMM is more accurate than Rule-based Classifiers.• Kinect is the best device to use due to infrared feature.• Kinect V2 is better than Kinect V1 in Joint detection.• Kinect V2 has face Joints property over Kinect V1.
![Page 73: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/73.jpg)
Future Work• Interviewing Enhancement• Try other classifiers seeking better accuracy• Provide the Grammar Builder with more alternatives
![Page 74: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/74.jpg)
Sponsorship
![Page 75: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/75.jpg)
Presentation Demo
![Page 76: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/76.jpg)
Interview Demo
![Page 77: Kinect virtual-learning (SIM U KIN Graduation Project)](https://reader035.fdocuments.in/reader035/viewer/2022062420/55c9d0b4bb61eb1c718b4577/html5/thumbnails/77.jpg)