Kinect Arabic Interfaced Drawing Application

By : Mohammed Hethnawi Mohammed Zaza

Subhi Mara’ba Yasser Hisham

Contents Abstract Introduction Kinect Skeleton Gesture Capture Bitmap Implementation Combining Bitmap Drawer with Kinect Gesture

Capture Future Work Summary and Conclusion

Abstract Introduces brief history about Kinect. Defines Kinect sensors. Presents a brief info about Kinect Drivers

OpenNI Microsoft SDK beta 2

Provides a brief explanation of our bitmap helper class implementation

Presents our techniques of getting the advantage of capturing user’s gestures for implementing our drawer

Reveal early thoughts of our future work

Introduction Brief history about Kinect :

The technology behind Kinect was invented in 2005 by Zeev Zalevsky, Alexander Shpunt, Aviad Maizels and Javier Garcia.

In November 2010, Adafruit Industries offered an open-source driver for Kinect .

In December 2010, PrimeSense, released their own open source drivers ( which includes OpenNI organisation ) along with motion tracking middleware called NITE.

On June 16, 2011, Microsoft announced its official release of its SDK for non-commercial use.

Introduction Kinect sensors :

The Kinect sensor is a horizontal bar connected to a small base with a motorized pivot and is designed to be positioned lengthwise above or below the video display .

The device features an “ RGB camera, depth sensor and multi-array microphone running proprietary software ”, which provide full-body 3D motion capture, facial recognition and voice recognition capabilities .

Introduction Kinect Drivers : OpenNI (Open Natural Interaction ):

It’s an industry-led, not for profit organization focused on certifying and improving interoperability of natural interaction devices.

Natural Interaction Devices are devices that capture body movements and sounds to allow for a more natural interaction of users with computers.

The OpenNI framework provides a set of open source APIs , which provide support for Voice and voice command recognition , Hand gestures , and Body Motion Tracking

Introduction Kinect Drivers : Microsoft SDK beta 2 :

It includes Windows 7 compatible PC drivers for Kinect device. It provides Kinect capabilities to developers to build applications with C++,

C#, or Visual Basic by using Microsoft Visual Studio 2010. It includes the following features : raw sensor streams, Skeletal tracking,

Advanced audio capabilities.

Tutorial

This application allows the user to draw on his canvas using his body gestures. The user has the choice to select a brush of his choice with a certain font size. The user also has the ability to choose the color he wants from a list of colors. The user can as well save his work and review it later.

The user has to raise his left hand above his shoulder and moves the cursor with his right hand.

Tutorial(continue) The user has to lower his left hand below his

shoulder to start painting.

Bitmap implementation The drawing application provides four different

brushes With different brush font sizes:

1- eraser 2- airbrush 3- pen 4- brush;

We used a linear interpolation function to implement drawing bitmaps between line segments.

To destinguish between different brushes we used different number of steps in each brush interbolation function.

Cursor implementation

The first challenge we faced was that the Kinect lacks the feature of mouse click.

Thus we used the principle of hovering over regions for a certain amount of time instead to implement mouse click.

We extended the C# given buttons and check lists classes with hovering methods to facilitate the use of Kinect

We introduced a focused regions for each button that hovering over enable it ( cause and event such as ischecked … etc)

Kinect Curser Instead of using joints movement as indicators

of certain actions we used joints coordinates. We implemented the mouse movements by

replacing the mouse coordinate with the right hand joint coordintaes (x,y) thus making the cursor trace the right hand movements.

“Nui.Vector nuiv = skeleton.Joints[Nui.JointID.HandRight].ScaleTo((int)ActualWidth, (int)ActualHeight, 0.50f, 0.30f).Position;

CursorPosition = new Point(nuiv.X, nuiv.Y);”

Kinect Curser (continue)

We also used the feature of coordinates as an action as we introduced left hand joint y coordinate and the left shoulder joint y coordinate and compared between them as a threshold for virtiualized mouse click.

“Nui.Joint lh = skeleton.Joints[Nui.JointID.HandLeft];

Nui.Joint ls = skeleton.Joints[Nui.JointID.ShoulderLeft];

bool isdown = lh.Position.Y < ls.Position.Y;”

Kinect Curser (continue)

It should also be mentioned that our application runs on kinect if it is available and if not it can use the mouse for the cursor coordinates.

“Loaded += (s, e) =>

{

if (MainWindow.Instance == null || MainWindow.Instance.NuiRuntime == null)

CompositionTarget.Rendering += (s2, e2) => OnUpdate(null, new Nui.SkeletonFrameReadyEventArgs());

else

MainWindow.Instance.NuiRuntime.SkeletonFrameReady += OnUpdate;

};”

Further work

We will divide our further work into three main fields.

1- Develop the drawing project and add more features to it.

2- Develop the keyboard simulator (get things kinectified !! Why NOT )

3- Introducing an advanced program to translate the Deaf language.

Developing the drawing application

This application is a simple drawing app. However we are planning to add more utilities such as:

1- More brushes. 2- Auto complete and auto correct utility. 3- Simplify the drawing techniques. (two

handed drawing). 4- Add more standard shapes.

Getting things kinectified !! Why NOT

Here we introduce an interesting idea about getting any software kinectified.

Keyboard simulator is a smart interfacing software that shows you a list of the running programs all you have to do is just picking one and start sending characters to it by Kinect gesture detector.

Deaf Language Translator ( DLT )

This software is not developed yet in any part of the world, the idea is to translate the speechless moves using the kinect gesture detector and then tells us what each move means.

Sounds good and Humanist but very hard to implement since we have to define each move and we cant use the already implemented moves !!

Summary and Conclusion

Kinect technology is still under development which made our task harder to implement.

We faced many challenges during the process such as: Having to program using xml and C# which have no previous experience in.

However we got the chance to get introduced to them which is beneficial for us as IT engineers.

Implementing the bitmap class was one of the hardest tasks. We used many resources and bitmap classes to implement.

After all this was one interesting and fun to do project ( hopefully prebaring us for our graduation project).

References

Microsoft SDK samples OpenNI samples Microsoft SDK beta 2 documentation www.wikipedia.com Our website : http://alrasam.0fees.net/about.html