By Rishabh Maheshwari
description
Transcript of By Rishabh Maheshwari
![Page 1: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/1.jpg)
ByRishabh Maheshwari
Kinect AppDevelopment
![Page 2: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/2.jpg)
Objective of today’s lecturePlay Angry Birds in 3D
![Page 3: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/3.jpg)
What’s an image ?An image is simply a collection of pixels, each of which contains some data.( A pixel is characterized by (x,y) )
Let us see some types of images...
![Page 4: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/4.jpg)
Binary ImageEach Pixel has either 1 (White) or 0 (Black)
Each pixel has 1 bit information( Binary images are seldom used )
0 0 0 0 0 0 0
0 0 1 1 1 0 0
0 0 1 1 1 0 0
0 0 1 1 1 0 0
0 0 1 1 1 0 0
0 0 0 0 0 0 0
![Page 5: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/5.jpg)
Grayscale
Each Pixel has a value from 0 to 255.0 : black and 255 : White
Between 0 and 255 are shades of b&w.
Each pixel has 1 byte informationIt is stored as an array of bytes.
![Page 6: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/6.jpg)
Grayscale Image
![Page 7: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/7.jpg)
RGB Image
Each Pixel stores 3 values :-R : 0- 255G: 0 -255B : 0-255
Each pixel has 3 bytes of informationIt is also stored as an array of bytes.
![Page 8: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/8.jpg)
RGB image
![Page 9: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/9.jpg)
Before moving to depth image, we must familiarize ourselves with the basics of kinect.
What is a kinect camera ? Kinect is a camera which gives R , G , B and depth information of each pixel.
![Page 10: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/10.jpg)
How does Kinect work?Kinect has 3 components :-• color camera ( takes RGB values) • IR camera ( takes depth data )• Microphone array ( for speech recognition )
![Page 11: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/11.jpg)
Depth Image
![Page 12: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/12.jpg)
But what’s the use of Depth ??
For this, Lets Discuss someImage Processing
![Page 13: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/13.jpg)
Background/Foreground Subtraction
1. On an Image ( Pixel Select Method )
2. On a Running Video ( Running Average Method )
![Page 14: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/14.jpg)
Edge Detection (the gradient)
![Page 15: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/15.jpg)
How can a Depth Image help in the above two ??
Back to kinect ….
![Page 16: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/16.jpg)
Player A player is the (human) skeleton which is detected by kinect sdk. There can be multiple players. Each pixel stores the corresponding “player index”.
Player index = 1
Player index = 2
By default:-Player index = 0
![Page 17: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/17.jpg)
Depth Image (Specific To Kinect sdk v1)
Each pixel stores :-• Player index : 0 – 7 ( 3 bits )• Depth( in mm) : 0 – 8192 ( 13 bits )
It is stored as an array of shorts.( A short is a 16 bit data type)
P P P D D D D D D D D D D D D D
Player Index Depth
![Page 18: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/18.jpg)
Some important datatypes:-• Kinect is defines as a datatype ( same as int or char) KinectSensor _kinect;
• Kinect sdk can handle multiple kinects at same time and treats these kinects as an array of kinect datatype :-
_kinect =KinectSensor.KinectSensors[0];
• DepthImagePoint is a struct which stores X , Y and Depth of a point :-
DepthImagePoint xyz;You can use: xyz.X xyz.Y xyz.Depth
![Page 19: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/19.jpg)
Kinect has 3 streams
• ColorStream : contains RGB data as byte array
• DepthStream : contains depth data as short array
• SkeletonStream : a template (What ??)
You need to enable these streams in the beginning as per your requirements.
![Page 20: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/20.jpg)
What is a SkeletonStream ?
When skeletonstream is called , it recognizes skeletons and populates pixels of depthstream with player index.
*If skeletonstream is not enabled, player index of all pixels of depthstream will remain 0.
![Page 21: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/21.jpg)
Using skeletonstream, kinect sdk provides us with 20 joints.Joints
![Page 22: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/22.jpg)
Eg:-JointType.HandRightJointType.FootLeftJointType.ShoulderLeftand so on....
Psuedo Code :-righthand = JointType.HandRightfloat x = righthand.X
Joints
![Page 23: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/23.jpg)
Let’s start with coding
1. Install visual studio.2. Install kinect sdk for visual studio.3. Select New Project4. In C# projects, select WPF project5. Add Microsoft.Kinect in reference of your
project.6. Write using Microsoft.Kinect;
![Page 24: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/24.jpg)
As you open your new project, a default window is provided.
There are 2 events associated with this window:-Window_Loaded() // when window loadsWindow_Closing() // When is pressed
![Page 25: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/25.jpg)
The Final basic code:-KinectSensor _kinect;
Window_Loaded(){
_kinect = KinectSensor.KinectSensors[0];_kinect.ColorStream.Enable();_kinect.DepthStream.Enable();_kinect.SkeletonStream.Enable();_kinect.Start();
}
Window_Closing(){
_kinect.Stop();}
![Page 26: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/26.jpg)
Lets see the code to understand more about “frame events”
![Page 27: By Rishabh Maheshwari](https://reader036.fdocuments.in/reader036/viewer/2022062305/56816385550346895dd46e9c/html5/thumbnails/27.jpg)
Questions ?For online video lectures :-http://channel9.msdn.com/Series/KinectQuickstart