Sign Recognition Presentation- Sahil Narang

7/27/2019 Sign Recognition Presentation- Sahil Narang

1/18

Sign LanguageRecognition

SUBMITTED BY:PRATISH NAIR

SAHIL NARANG

SUSHANT BHASIN

TUSHAR GUPTA

Guide: Sushil Kumar


2/18

INTRODUCTION

The proposed system is capable of recognisingfinger-spelling hand shapes.

It will provide interface for keyboard-lessinteraction and for American sign languagelearning.

The system needs to be trained with respect tomultiple users, which involves creation of

database which is a collection of samples of fingerspelling by these users.

We propose an approach that can recognize withvery good accuracy across different users.

2


3/18

TECHNOLOGY USED

3

KINECT SENSOR

Kinect is a motion sensing inputdevice by Microsoft for WindowsPCs.

Based around a webcam-style it

enables users to control andinteract through a natural userinterface using gestures and spokencommands.


4/18

Libraries Used

Microsoft SDK 1.5 To acquire RGB and Depth Images

OpenCV 2.4.2 Basic Image Processing Tasks

Tiny Thread Multi-Threading Capabilities in C++

4


5/18


6/18

PROCEDURE

User makes sign in front of the camera

The hand is extracted from the frame using

Distance Threshold

A bounding Rectangle is generated aroundthe hand

Hand Contours are extracted

The largest contour is selected for processing

Mode chosen: Digit Recognition or ASLRecognition

6


7/18

Digit Recognition Mode

Approximating the hand contours with a polygon

Finding the Convex hull

Detection of convex and concave points

Filtering of convex and concave points

Identifying Digit7


8/18

8

0 1 2-5

Concave Points 0 1 1+

Convex Points 0+ 1 (outside) =digit

Predicting Digits


9/18

Concave Points: 5 (+1)

Convex Points: 5

This is case 3. So, Predicted Digit=59


10/18

ASL Recognition Mode

10

Rotate the contour so that its angle ofinclination with bounding box is zero

Scale the image so that both images i.e.

current image and database image are ofsame size

Find Image Moment using the formula

Where A is the first image, B is the second imageand m=sign(h)*log h ,where h is the HU moment ofimage


11/18

11

Find Ratio using the formula

Ratio = (No. of pixels that have similarvalue) / (Total no. of pixels)

If the moment value is below

Imagemoment_thresholdand ratio is greater

than ratiomin we have a match


12/18

Digit Recognition Results

12

0 1 2 3 4 50 15

1 15

2 1 14

3 1 4 8 2

4 1 1 13

5 1 6 8

A

C

T

U

A

L

PREDICTED


13/18

ASL Recognition Results

13

i.PREDICTED

A B D K L W Y NULL

A 8 2

B 2 6 2

D 10

K 8 2

L 6 4

W 2 4 4

Y 1 8 1

A

C

T

U

A

L


14/18

LITERATURE SURVEY

Real-time Sign Language Letter and Word Recognition from DepthData by Uebersax, Gall, Bergh ,Gool.

A system for recognizing letters and finger-spelled words of theAmerican sign language (ASL) in real-time.

The system segments the hand and estimates the hand orientation

from captured depth data.

The letter classification is based on average neighbourhood

margin maximization and relies on the segmented depth data of

the hands.14


15/18

LITERATURE SURVEY

Combining RGB and ToF Cameras for Real-time 3D Hand GestureInteraction Michael Van den Bergh, Gool Proceedings of theIEEE(WACV 2011)Kona, Hawaii, January 2011

Time-of-Flight (ToF) and other IR-based cameras that registerdepth are used for finding depth information.

Furthermore, the depth information allows us to track the

position of the hand in 3D

The result is a real-time hand gesture interaction system that

allows for complex 3D gestures and is not disturbed by objects or

persons in the background. 15


16/18

REFERENCES

S. Mitra and T. Acharya. Gesture recognition: A survey. IEEETransactions on Systems Man and Cybernetics, Part C,37(3):311324, 2007.

E. Ong and R. Bowden. A boosted classifier tree for handshape detection. In Proc. of FGR, pages 889894, 2004

S. Ong and S. Ranganath. Automatic sign language analysis:A survey and the future beyond. IEEE Transactions onPattern Analysis and Machine Intelligence, 27(6):873891,2005.

Wikipedia. American manual alphabet.

http: //en.wikipedia.org/wiki/American_manual_alphabet

16


17/18

REFERENCES

M. Van den Bergh and L. Van Gool. Combining RGB

and ToF cameras for real-time 3D hand gesture

interaction. In Proceedings of the IEEE Workshop

on Applications of Computer Vision (WACV 2011),2011

S. Liwicki and M. Everingham. Automatic

recognition of fingerspelled words in British sign

language. In Proc. Of CVPR, pages 5057, 2009

17


18/18

THANK YOU

18

Sign Recognition Presentation- Sahil Narang

Documents

Transcript of Sign Recognition Presentation- Sahil Narang