Coding4Fun: Build Fun, Cool, Commercial Applications Using the Kinect for Windows SDK Dan FernandezDirectorMicrosoft Corporation

Rick BarrazaSenior Technical Evangelist

Microsoft Corporation

DEV330

AGENDA

Kinect OverviewManaging SensorsKinect Data – Color, Depth, SkeletonKinect StudioFace TrackingAudio Programming

KINECT IS TWO THINGS

HardwareKinect for Xbox 360Kinect for Windows

SoftwareKinect for Windows Runtime 1.5

For deploying Kinect applications

Kinect for Windows SDK 1.5API’s for Color, Depth, Skeletal tracking, and Audio

Developer Toolkit 1.5.1Face Tracking, Kinect Studio, Kinect Controls, Samples, etc.

KINECT SENSOR

IR Emitter Color SensorIR Depth Sensor

Tilt Motor

Microphone Array

KINECT CAMERA ANGLES

Horizontal

Vertica

l

KINECT API BASICS

Manage Kinect state ConnectedEnable Color, Depth, Skeleton, AudioStart Kinect Streams

Get DataEvents – AllFramesReadyPolling – OpenNextFrame

IMAGE PROCESSING 101

BGR32 Format – Every Pixel (0,0 | 0,1 | 0,2) has blue, green, red, empty

B G R Empty

255 255 255 0

Kinect ImageSizes: 80x60, 320x240, 640x480DPI: 96Stride: # of bytes per single line:

Width (320 or 640) x 4 bytes (B,G,R,E)

B G R Empty

255 255 255 0

320x240

640x480

80x60

CAMERA FUNDAMENTALS

DEMO

DEPTH DATA

Returns the distance and player index for every pixelEx: 320 x 240 = 76,800 pixels

DistanceDistance in mm from Kinect ex: 2,000mm (6.56 feet)

PlayerIdentify which player index is shown (1-6 or 0 for no player)

Depth is a 16-bit short D D D D D D D D D D D D D P P P

15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

Depth Player Index

DEFAULT/NEAR MODE RANGE

Near Default

0.4m/1.3 ft 0.8m/2.6 ft

DISTANCES – KINECT BETA

1.3’ 2.6’ 9.8’ 13.1’ 26.2’

.4 .8 3 4 8

DefaultMode

Near Mode

Feet

Meters

Unknown Normal

DISTANCES – KINECT V1+

1.3’ 2.6’ 9.8’ 13.1’ 26.2’

.4 .8 3 4 8

DefaultMode

Near Mode

Feet

Meters

Unknown Too Near Normal Too Far

DEPTH FORMULASDepth datashort[] allData = new short[depthFrame.PixelDataLength];

Get depth for a point (X:160, Y:120, Width: 320)int index = (width * y) + x; // 38,560

Get X, Y for an index (Index: 5,000, Width:320)int x = index % width; // 200int y = index / width; //15

Distance Formulashort depthPoint = allData[index];int depth = depthPoint >> DepthImageFrame.PlayerIndexBitmaskWidth; // 2,000

Player Formulaint player = depthPoint & DepthImageFrame.PlayerIndexBitmask; // 1

DEPTH DATA

DEMO

SKELETAL TRACKING

Max of two skeletons per Kinect6 Player Proposals

HandLeft

WristLeft

ShoulderLeft

Head

ElbowLeft

ShoulderRight

HandRight

WristRight

ElbowRight

ShoulderCenter

FootLeft

AnkleLeft

KneeLeft

HipLeft

HipCenter

HipRight

FootRight

AnkleRight

KneeRight

Spine

Seated (10)

Default (20)

COORDINATES

Z AxisX Axis

Y AxisJoints in meters from camera

Negative

Positive

Positive Negative

Positive

JOINT TRACKING & SMOOTHING

Each joint has associated tracking stateTracked, Not tracked, or InferredInferred - Occluded, clipped, or low confidence joints

Use TransformSmoothParameters to smooth joint data to reduce jitter

Kinect transform smoothing uses Holt Double Exponential Smoothing - http://en.wikipedia.org/wiki/Holt-Winters

Skeleton.ClippedEdges = FrameEdges.Top;

PHYSICAL INTERACTION ZONE

Fatigue Kills Gestures Use Coding4Fun ScaleTo extension method

SKELETAL TRACKING & KINECT STUDIO

DEMO

KINECT AS ANOFFICE MONITOR.

KINECT AS ANOFFICE MONITOR.

KINECT FOR MONITORING

DEMO

KINECT SERVICE – PROJECT DETROIT

http://download.codeplex.com/Download?ProjectName=kinectservice&DownloadId=336928

Kinect data over the network using socketsColor, Depth, Skeleton, and AudioClient libraries for WPF, Windows Phone & WinRT

FACE TRACKING

Real-time tracking of multiple faces2D/3D Mesh and Points3D Head PoseAnimation Units

Z Axis

3D HEAD POSE

Pitch

+20

-20

Yaw

-45 +45

Roll

-90 +90

FACE TRACKING

DEMO

SOUND POSITION

Sound Source Angle – The angle and confidence level of where audio is coming fromBeam Angle – The angle used to record audio that you can set as a “directional microphone”

SPEECH RECOGNITION GRAMMAR

<!-- Confirmation_Yes._value: string ["Yes"] --><rule id="Confirmation_Yes" scope="public"> <example> yes </example> <example> yes please </example> <one-of> <item> yes </item> <item> yeah </item> <item> yep </item> <item> ok </item> </one-of> <item repeat="0-1"> please </item> <tag> out._value = "Yes";</tag>

var grammar = new Choices();grammar.Add(“yes please");grammar.Add(“yes");grammar.Add(“yeah");grammar.Add(“yep");grammar.Add(“ok");

Ensure AutomaticGainControl = falseSample : C:\Program Files\Microsoft SDKs\Speech\v11.0\Samples\Sample Grammars

Speech Recognition: English/Great Britain, English/Ireland, English/Australia, English/New Zealand, English/Canada, French/France, French/Canada, Italian/Italy, Japanese/Japan, Spanish/Spain and Spanish/Mexico

VOICE-ENABLED TIC TAC TOE

DEMO

JOINT ORIENTATION

Rotation in camera space, or amount of rotation from parent bonePlayer orientation stored in HipCenter, or ShoulderCenter when seatedMatrix or Quaternion format for animations

BOXINGBOTS

DEMO

Related Content

BoxingBots – In Expo Hall

Resources

Connect. Share. Discuss.

http://europe.msteched.com

Learning

Microsoft Certification & Training Resources

www.microsoft.com/learning

TechNet

Resources for IT Professionals

http://microsoft.com/technet

Resources for Developers

http://microsoft.com/msdn

Evaluations

http://europe.msteched.com/sessions

Submit your evals online

© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS

PRESENTATION.