Final Report 21-4

Controlling Computer Using Hand Gesture Recognition

CHAPTER 1

INTRODUCTION

1.1 Introduction:

The use of hand gestures provides an attractive alternative to cumbersome

interface devices for human-computer interaction (HCI). In particular, visual

interpretation of hand gestures can help in achieving the ease and naturalness desired for

HCI. The goal of our project is to develop a real-time system capable of understanding

commands given by Hand gestures. The user must be able to communicate to computer,

all basic commands required by a human-computer interface.

Our system will provide following functions:

1: Mouse movement on screen according to the hand movement.

2: Right click, left click, double clicks of mouse.

3: Navigation key press which are up, down, left, right.

4: Number recognition from 0 to 5.

Another advantage is that the user not only can communicate from a distance, but

need have no physical contact with the computer. However, unlike audio commands, a

visual system would be preferable in noisy environments or in situations where sound

would cause a disturbance. The visual system chosen was the recognition of hand

gestures. The amount of computation required to process hand gestures is much greater

than that of the mechanical devices, however standard desktop computers are now quick

enough to make this project hand gesture recognition using computer vision — a viable

proposition. This interface is simple enough to be run using an ordinary webcam and

requires little training.

SCOE, Information Technology, 2010-11.

1


1.2 Literature Survey:

1) A simplification used in similar type of projects, which was not found in any

recognition methods researched, is the use of a wrist band to remove several degrees of

freedom. This enabled three new recognition methods to be devised. The recognition

frame rate achieved is comparable to most of the systems in existence (after allowance

for processor speed) but the number of different gestures recognized and the recognition

accuracy are amongst the best found. Figure shows several of the existing gesture

recognition systems along with recognition statistics and method.

Table 1.0: Literature survey

Paper Primary method of recognition

Number of gestures recognized

Background of gesture images

Additional markers required e.g. wrist band

Number of training images

Accuracy

Frame rate

[Bauer & Hienz, 2000]

Hidden Markov Models

97 General Multi-colored Gloves

7-hours signing

91.7% -

[Starner, Weaver & Pentland, 1998]

Hidden Markov Models

40 General No 400 training sentences

97.6% 10

[Davis & Shah, 1994]

Finate state machine/model matching

7 Static Markers on glove

10 sequences of 200 frames

98% 10


2


2) Enabling Simple Window Management Using Pinch Gesture Recognition by

Benjamin Baldus, Debra Lauterbach, Juan Lizarraga: In this project, they developed a

machine-user interface which implements pinch gesture recognition using simple

computer vision techniques. The interface allows the user to use their hands to control the

mouse as well as easily move and resize windows that are open on their screen. This

interface is simple enough to be run using an ordinary webcam and requires little

training.

One task which is especially cumbersome to perform with the mouse is moving

and resizing open windows. This is something that many computer users do quite often -

opening several applications, folders, browser windows, etc. on their screen, and then

switching between them frequently. But, on most computers, the only way to move a

window is by clicking and dragging it by its header/title bar, and the only way to resize is

by clicking and dragging on the bottom-right corner. This can be annoying to do for

managing multiple windows, which is the problem that this application aims to solve.

Fig shows screenshots of their projects.


3


Fig.1.2 Window management system

These are further more paper published related to this project.

1) Hand Gesture Recognition Using Computer Vision By -Ray Lockton, Balliol

College, Oxford University

Identify the letters and number based on the American one-handed sign

language.

Perform mouse operations


4


Text editor

2) Interprets your Hand Gestures By Ashutosh Saxena, Aditya Awasthi, Indian

Institute of Technology Kanpur.

perform mouse operation

1.3 Project Plan :

Plan of our project is given below. 1st part of plan contains whole image

processing. There are lots of steps to be performed in image processing given below. 2nd

part contains only gesture implementation part which uses system programming for

implementation as shown below.

Part 1:

Image processing

Video capturing from camera.

Detecting wristband from image.

Calculating wrist band to hand ratio.

Calculating region of interest.

Skin pixel detection.

Hand segmentation.

Hand detection and identification of gestures.

Part 2:

Gesture implementation

Perform mouse operations depending on hand gestures.

Perform few keyboard operations like navigation keys,

numbering from (0 to 5).


5


Block diagram of project is given below:

Fig.1.3 Block Diagram

Block diagram of this project shown above has six blocks like capture image,

detect wristband, hand segmentation, hand detection, gesture detection and gesture

implementation etc.

Capture image is nothing but capturing video input through camera. This input is

given for image processing in further steps. If camera quality is better, captured video is

also better and results to be achieved are also better and efficient. This captured video is

given to next step called as detect wrist band. While using this system user should have

wrist band tied with his/her hand. Without wrist band system is not going to work. Detect

wrist band step scans whole image and selects region of interest where wrist band is

present. Only region of interest of wrist band is detected here no other processing is done.


6


Next step is hand segmentation. In this step region of interest above the wrist

band where probable hand is present is determined and only that much area is used for

further processing. Skin pixels present in that area is detected, which are nothing but skin

pixels of our hand. In this way hand is segmented. For every people wrist band to hand

ratio is approximately constant so it is beneficial to us while segmenting hand.

Segmented hand is given for hand detection technique. Here two techniques

namely convex hull and convex defect are used for actual detection of hand. Convex hull

technique joins extreme points of polygon which are present in certain area. In our

project convex hull technique joins points which are present on tip of every finger with

each other and to the base of palm. Furthermore convex defect technique calculates actual

defects present between fingers. In this way actual hand is detected.

Detected hand is given for further processing. Gesture detection technique takes

this hand and identifies gestures. Detected gesture could be related to left click, right

click or number counting from 0 to 5.

After identifying gestures, are implemented in gesture implementation technique.

Here some amount of system level programming is required for implementing gestures.

For implementing mouse operations, mouse events are needed to be captured. In this way

our system is implemented for specified operations.


7


1.3.1 Project Development Life Cycle:

Waterfall model we are going to use for this project implementation.

Fig 1.3.1 Waterfall model

Waterfall model is the most well known software lifecycle development

model. It is very simple to understand and use. Each next phase in this model

must beginning only after the previous phase is over.

Following are Phases of This Waterfall model:

a) Requirement Analysis and Project Planning:

In this first stage of waterfall model is to understand the requirements.

When the requirements have been noted, it is important to make sure that the


8


requirements are detailed and accurate and there is no place for any ambiguity.

Before analysis, requirement is gathered from various sources. The sources of

requirements could be anything like internet or any other resources.

After gathering and analysis of requirements project planning is carried

out. Project planning includes actual plan of project including stepwise working

of project and estimation of total cost along with estimation of maintenance cost.

Project planning is done in such a way that project should be cost effective and

efficient.

b) System Design:

The requirements that are gathered in the previous phase are broken down

into logical units, so that the software process becomes easy for implementation.

This is the stage, when the software requirements along with the hardware

requirements for every unit are identified. Then the designs are made accordingly.

Design of system is prepared after whole study of actual plan. Design is

nothing but implementation of plan in rough model which is further refined in

next phases of water fall model. Design of system gives actual idea of system,

how system will look, how it will perform actions etc.

c) System Implementation:

In this phase the actual development of the software takes place. This

phase is also known as coding and verification phase. Based on the algorithms

written in the previous phase, software program is written.

System is implemented with help of coding. Coding can be done in any

language which depends upon developer’s choice and efficiency of coding


9

http://www.buzzle.com/editorials/8-31-2004-58670.asp


language to yield best software. Verification phase verifies code of system for

errors. It checks whether lines of code are sufficient or not. In this way system is

implemented with the help of coding and verification. System is implemented in

various modules as modules are small and easy to implement compared to whole

system at a time.

d) System Integration and Testing:

In this phase all the modules are integrated, after which the software is

tested for correct output. All the bugs introduced due to integration are removed.

There are number of techniques are used for integration and testing depending on

efficiency of technique. Testing techniques used here are unit testing for

individual module, integration testing for integrated modules and system testing

for testing whole system.

If there is any bug in the software, it is removed while testing. Tested

software is given for next phase i.e. deployment phase of water fall model.

e) System Deployment and Maintenance:

This makes for the final phase of the waterfall model, where the software

is deployed at the user side, after it has undergone thorough testing. After the

deployment of the software, routine maintenance work is carried out. In

maintenance phase software is repaired if any problem occurs and returned back

to user. There must be maintenance phase after delivery of software as it

beneficial for maintaining good relationship with customers.

Following are some advantages of Waterfall model

Simple and easy to use.


10


Easy to manage due to the rigidity of the model – each phase has specific

deliverables and a review process.

Phases are processed and completed one at a time.

Works well for smaller projects where requirements are very well

understood.


11


CHAPTER 2

REQUIREMENT GATHERING

2.1 Problem Statement :

“Controlling Computer Using Hand Gesture Recognition ”

Basic purpose is to develop system such that we are able to operate computer

with the help of hand gestures without using mouse. This project helps us in introducing

new computer interface, provided interface will be more natural and user friendly. It also

reduces physical contact with devices and minimizes hardware required for interfacing.

In this project wrist band of single color is used. We first detect wristband using

camera, and calculates region of interest. Wrist band helps in removing background.

Camera detects skin color and segment hand from background image. Finally hand

gesture is identified and respective operations are performed on computer.


12


Fig.2.1 Gesture recognition system

2.2 Vision:

2.2.1 Background and Basics:

Controlling Computer using Hand Gesture Recognition is a kind of image

processing and research oriented project. It also contains computer vision and

multimedia. In this project we are going to use webcam for capturing videos as

hardware. Any camera that supports streaming video input can be used. No other

external hardware is required. We implement this project in C with the help of

editor DevC++ and OpenCV libraries. OpenCV libraries are special kind of

computer vision libraries which help in capturing videos using webcam easily and

storing images. DevC++ is configured in order to use OpenCV libraries.

WindowsXP OS is used for operations.


13


A visual system would be preferable in noisy environments or in

situations where sound would cause a disturbance. The visual system chosen was

the recognition of hand gestures. The amount of computation required to process

hand gestures is much greater than that of the mechanical devices, however

standard desktop computers are now quick enough to make this project hand

gesture recognition using computer vision — a viable proposition.

This project helps us in introducing new computer interface, provided

interface will be more natural and user friendly. It also reduces physical contact

with devices and minimizes hardware required for interfacing.

2.2.2 Scope:

1. Capturing video input from camera:

Webcam with desirable quality is used for capturing input. We have

written code using OpenCV library to capture images from camera. Here Dev C+

+ editor is used for editing and running program. cvCaptureFromCAM() and

cvQueryFrame() are OpenCV functions used to capture images from camera.

2. Detection of wristband:

User has to use wrist band while using this system. Without wristband it is

impossible to use system. Wrist band of yellow color with width of 4cm is used.

This project first detects wrist band and finds region of interest of wrist band.

Wrist band helps in easily segmenting hand from background image.


14


3. Segmentation of hand from background image:

Probable area of hand is determined using wrist band. Only probable area

of hand is scanned for next step. cvRectangle() is OpenCv function to determine

region of interest. Skin detection technique is used within region of interest so that

actual hand is detected. Only region of interest is considered and remaining

portion of image is ignored. In this way processing region is reduced and

processing time is saved here. This segmented hand is given for further

processing.

4. Recognize the hand gesture.

In above step actual hand is not detected. Convex hull and convex defects

techniques are used to calculate exact hand. Convex hull technique calculates

contour of hand including finger tips. Convex defect technique used to calculate

defects in between fingers. In this way actual hand is traced out.

5. Implementation of gestures:

After detecting hand gestures, are implemented with the help of

windows.h file of system. Some amount of system level programming is required

to be implemented here. Performing the PC control activities depending upon

gesture recognized such as:






15


2.2.3Requirements:

a) Related Hardware Requirements: Computer and Webcam.

Computer:

Any standard PC running Windows XP is capable of running our project.

The minimum hardware requirements are those of the operating system and the

required software.

Webcam:

We used a standard USB webcam of desired resolution. This viewing

area proved to be sufficient, though a larger area would have advantages as far as

having more room for performing gestures. Webcam with maximum resolution

will be more efficient. You can use webcam of higher resolution. Webcam should

be in front of user that is parallel to screen pointing towards user.

b) Operating Systems:

WindowsXP OS:

This operating system developed by Microsoft is a stable and trustworthy

platform for running applications made with Dev C++.


16


c) Availability of resources: OpenCV Libraries and DevC++ .

OpenCV Libraries:

It is a C++ based computer vision library developed by Intel. It focuses

mainly on real-time image processing, and gives the user a large amount of tools

that provide extensive functionality on this field. In our project we used OpenCV

for capturing frames from the webcam and processing the images using

morphology Monitor.

cvBlobsLib is a library for OpenCV which performs binary image

connected components labeling. It also has additional functionalities such as

manipulating, filtering, and extracting blob features. This library is used in our

project to determine when a pinch is being made, and to determine the center of

the blob for moving the mouse.

Dev C++:

Dev C++ is the main editor used for coding purpose. In order to use

OpenCV library, we have configured OpenCV with DevC++. Configuration

includes adding library paths for .lib files, .bin files, include files for both C as

well as C++.

Latest version of Dev C++ is used that is Dev C++ 4.9.9.2.

d) Network requirement :

This project is performed on single stand alone machine. So there is no

need of networks.


17


e)Other Requirements :

Yellow color Wrist Band:

We have used yellow color wrist band of width approximately 4 cm. This

band is helpful for segmenting hand from background image. This band avoids

extra work needed for subtracting background from foreground. In this way it

reduces extra efforts and adds flexibility to our projects.

For best use of this system we must require good camera and user with at

least a hand. Here we have defined some conventions for operations such as left-

click, right-click etc. Depending on gestures recognized specific operations are

performed on computer.

System should easily capture hand images and should give quick response

to the hand gestures to perform respective operations. It should support maximum

requirements of user. It should provide better flexibility to the user. It should have

better efficiency and minimum drawbacks.

System provides following services to user:






18


2.3 System Requirement Specification

2.3.1 Introduction

2.3.1.1.Problem Statement:


2.3.1.2. Describe the Scope of the System:

1. Capturing video input from camera:

Webcam with desirable quality is used for capturing input. We have

written code using OpenCV library to capture images from camera. Here Dev C+

+ editor is used for editing and running program. cvCaptureFromCAM() and

cvQueryFrame() are OpenCV functions used to capture images from camera.

2. Detection of wristband:

User has to use wrist band while using this system. Without wristband it is

impossible to use system. Wrist band of yellow color with width of 4cm is used.

This project detects wrist band and finds region of interest of wrist band. Wrist

band helps in easily segmenting hand from background image.

3. Segmentation of hand from background image:

Probable area of hand is determined using wrist band. Only probable area

of hand is scanned for next step. cvRectangle() is OpenCV function to determine

region of interest. Skin detection technique is used within region of interest so that

actual hand is detected. Only region of interest is considered and remaining

portion of image is ignored. In this way processing region is reduced and


19


processing time is saved here. This segmented hand is given for further

processing.

4. Recognize the hand gesture.

In above step actual hand is not detected. Convex hull and convex defects

techniques are used to calculate exact hand. Convex hull technique calculates

contour of hand including finger tips. Convex defect technique used to calculate

defects in between fingers. In this way actual hand is traced out.

5. Implementation:

After detecting hand gestures, are implemented with the help of system

level programming which contains windows.h file of system. Some amount of

system level programming is required to be implemented here. Performing the

PC control activities depending upon gesture recognized such as:






20


2.3.2. Proposed System (Project)

2.3.2.1 System Overview :

Project requires stand alone PC with windows XP installed on it. Because

XP performs smooth operations, it is flexible and supports maximum of functions.

Webcam is required for capturing video input. Softwares required are OpenCV

library and Dev C++ editor. OpenCV is computer vision library which provides

all functionality related to vision of computer and image processing. Wrist band is

another requirement of system.

The detailed overview of this project is given below:

1. Capturing the live video input from camera:

Camera used can be a simple webcam. We need to capture the live feed of

camera. One solution to this can be OpenCV. We can use OpenCV library which

is supported on common operating systems like Windows and Linux. Captured

video is given for processing. Program is written using OpenCV libraries.

OpenCV library has lots of functions related to image processing and GUI

programming. Camera quality affects lots of things in processing.


21


Fig.2.3.2a Captured image through camera

2. Detection of wrist band:

Wrist band detection is crucial step in this project. It has many advantages

over other techniques. A yellow color wristband is used in this project. As name

suggests wrist band is tied with wrist. Captured image is scanned for identifying

band. If there is no band present in image, system is not going to work properly.

Wrist band helps in removing background image from foreground image.

Background subtraction is very difficult task in this project, which is made easier

by wrist band detection technique.

Another advantage of wrist band is background of variant color also has

no effect on system performance. If band gets detected properly, then system will


22


run properly. But there is a disadvantage of this system, if background contains

color similar to wrist band then there is a problem while processing the image.

Screen shot below shows how wristband is detected, region of interest of

wrist band and probable region of hand above the wrist band.

Fig.2.3.2b) Wristband detection


23


3. Segmentation of hand:

Before we talk about the segmentation of hand from background we need

to consider the type of background used. It can be

a)Constant: Considering the plane background, segmentation of hand

from background is comparatively easy. It would be far easy to segment hand if

the background color is different from skin color.

b)Varying: In varying background segmenting hand is quite tedious job.

Because you not only need to recognize the hand but have to remove all

background objects which resembles to skin color.

Here in this project, background subtraction becomes easy as we are using

wrist band to detect hand. Using wrist band background of any color can be easily

removed but it fails when background has same color with wristband or skin.

After detecting wrist band, wrist band to hand ratio is calculated for

determining region of interest of hand. In this region of interest (ROI) skin pixel

detection technique is used in order to detect hand. Only region of interest is

scanned in further phases. The R-G-B values of the skin can be used to separate

out pixels of skin from the background.

OpenCV libraries provide number of functions for detecting skin pixels

from image. In this way actual hand is segmented from whole image in order to

reduce processing.


24


4. Recognizing hand:

There are number of methods to recognize hand, but we have used

wristband technique to recognize hand. Firstly wrist band is detected depending

on its color, region of interest of wrist band is calculated. We have used wrist

band to hand ratio to detect probable hand from image. Skin color separates hand

from background image. Skin color plays important role in detecting hand. The R-

G-B values of the skin can be used to separate out pixels of skin from the

background.

There are two techniques to detect exact hand from ROI. These are convex

hull and convex defect. Convex hull technique joins extreme points of hand

including finger tips. It forms polygon of hand whose points are connected.

Convex defect technique finds actual defects in between fingers to give exact

image of hand. After getting exact image of hand we fill that image with color

which is less used in environment.

Screen shot below shows segmented hand, ROI of wrist band along with

separated hand from whole background.


25


Fig.2.3.2c Hand detection

5. Identify the gesture:

Gesture identification may or may not need the gesture database,

depending upon the algorithm used for identification. Here in this project we have

used convex hull and convex defect techniques to segment and detect hand

correctly. Convex hull and convex defect techniques determine exact portion of

hand which required for us to perform operations. In this way these techniques

helps in identifying hand gesture.

If current hand gesture is different from previous then gesture is detected.

We have predefined some gestures in the program.


26


Following figure shows how gestures are identified? In figure identified

gesture is number 4 as four fingers are straight.

Fig.2.3.2d Gesture identification


27


6. Implementing the gesture:

After recognizing the gesture we can evaluate its meaning and do the

action which is meant to be done. Some amount of system level programming is

required to be implemented here. Performing the PC control activities depending

upon gesture recognized such as cursor movements like up, down, left and right

etc. left click, right click, double click and numbers counted from 0 to 5.

1. Moving Mouse Cursor:

We used the wrist band center as a cursor controller to control mouse

cursor. We used the following approach for moving the mouse cursor. The

method is mapping cursor control. It means that the center of wrist band on a

camera screen can position maps to a desktop screen position. In other words, the

mouse cursor is placed on the desktop window along with the center of wrist band

position displayed on the camera screen position.

This method has a problem. If the resolution of the desktop window has a

higher resolution than the camera resolution, then the cursor position cannot be

accurate because when the camera resolution converts to the desktop window

resolution we lose intermediate value. We expect the ratio of jumping pixel is up

to 4 pixels. Another problem is when any one of the finger goes outside the

camera boundary then gesture is not identified properly. Cursor gets moved

according to center of wrist band only if index finger is straight and all other

fingers are curled. This method gives more accurate results.


28


2. Left Clicking and Double-Clicking:

To call system event for left clicking, at least two convex hull vertexes

have to be off the palm area which was computed in the previous part. In addition,

the x position of one of two vertexes should be lower than the other to restrict

detection of other finger tips. When the degree of the index finger and thumb is 70

to 90 degree then we can recognize that the gesture is left clicking. Actually, if the

thumb is placed outside the circle of the hand region, then the gesture is left

clicking. The double-clicking occurs when the thumb moves 0 to 90 degree and

back two times fast.

3. Right Clicking:

We simply implemented this part using two fingers as index finger and

middle finger. If both fingers are straight and separated by small space then

system performs right click operation. Left click and right click are performed if

count of fingers is two. But how does system differentiate between them? We

used a method in which distance between tip of fingers and angle between them is

calculated. If distance is large and angle is acute then left click is performed

otherwise right click is performed.

4. Recognizing numbers from 0 to 5:

We have implemented this with the help of number of straight fingers. If

there are two straight fingers then count is 2, three straight fingers then count is 3

and so on.


29


Figure given below shows, how numbers are recognized? As there are

three fingers straight so count is 3. In this way numbers are recognized from 0 to

5 using hand gesture recognition.

Fig.2.3.2e Recognition of numbers


30


2.3.3. Project Status

A) Classification :

The project can be classified into modules as:

1. Wrist band detection2. Hand segmentation3. Hand Detection4. Gesture identification and implementation.

B) Definition :

Basic purpose is to develop system such that we are able to operate

computer with the help of hand gestures without using mouse. This project helps

us in introducing new computer interface, provided interface will be more natural

and user friendly. It also reduces physical contact with devices and minimizes

hardware required for interfacing.

In this project wrist band of single color is used. We first detect wristband

using camera, and calculates region of interest. Wrist band helps in removing

background. Camera detects skin color and segment hand from background

image. Finally hand gesture is identified and respective operations are performed

on computer.

C) Responsibilities:

For best use of this system we must require good camera, wrist band and

user with at least a hand. Here we have defined some conventions for operations

such as left-click, right-click etc. depending on gestures recognized specific

operations are performed on computer.


31


System should easily capture hand images and should give quick response

to the hand gestures to perform respective operations. It should support maximum

requirements of user. It should provide better flexibility to the user. It should have

better efficiency and minimum drawbacks.

System provides following services to user:






32


D) Composition

1. Design [ UML Diagrams ]:

Use case Diagram:

System

System

User

Switch on/ off web cam

Makes guestures

initiates different operations

Valid guestures<<include>>

Valid operations

<<include>>

Machine

Checks whether cam is on

Detect Wrist Band

Detect Region of Interest

Detect Hand

Identify gesture

Perform Operation

Fig.2.3.3a Use case Diagram


33


Class Diagram:

Machine

+int cam+int fun+int temp

+int check()+int WBdetect(int color)+int hdetect()+void improcess()+pform_operation()

User

+char name+int hand+int temp1

+int cam()+void makegesture()

gives input to

Fig.2.3.3b Class Diagram


34


Sequence Diagram:

ALtsd

User Machine

1 : power on cam()

2 : Cam on()

3 : Moves hand to cam()

4 : Detect Wrist Band()

5 : Detect Region Of Interest()

6 : Detect Hand()

7 : Make gestures()

8 : Capture image()

9 : Process image()

10 : Identify Gesture()

11 : Perform operation()

12 : Error()

13 : Displays error message()

Fig.2.3.3c Sequence Diagram


35


Activity Diagram:

User Machine

Switch on cam Cam switched on

Moves hand to cam Detects Wrist Band

Detect region of Interest

Makes gestures Captures image

processes image

Identify gesture

Perform Operation Report error

[gesture identified] [gesture not identified]

Detect hand

Fig.2.3.3d Activity Diagram


36


DFD:

DFD Level 0:

Fig.2.3.3e DFD Level 0


37


DFD Level 1:

Fig.2.3.3f DFD Level 1

CHAPTER 3SCOE, Information Technology, 2010-11.

38


PROJECT ANALYSIS

3.1 Use-case Model :

System

System

User

Switch on/ off web cam

Makes guestures

initiates different operations

Valid guestures<<include>>

Valid operations

<<include>>

Machine

Checks whether cam is on

Detect Wrist Band

Detect Region of Interest

Detect Hand

Identify gesture

Perform Operation

Fig.3.1 Use case Diagram

3.2 Use case Specification/Document-Scenario description:


39


3.2.1 Use case specification : Make gesture

1. Description

Use case allows user to make gestures.

2. Actor

User

3. Flow of event

3.1Basic flow-

3.1.1 Start the application

3.1.2 Make the gestures

3.2Alternate flow

3.2.1 Make different gesture

4. Special Requirements:

Camera is installed properly.

5. Precondition

5.1 Camera is on

5.2 Wrist band must be used.

6. Post condition

6.1 Gestures made successfully.

3.2.2 Use case specification: Detect handSCOE, Information Technology, 2010-11.

40


1. Description

Use case allows machine to detect hand.

2. Actor

Machine

3. Flow of event

3.1Basic flow-

3.1.1 Switch on the camera

3.1.2 Capture gesture image

3.1.3 Detect wrist band.

3.1.4 Detect Hand.



5. Precondition

5.1 Camera is on


6. Post condition

6.1 Hand detected successfully.

3.2.3 Use case specification: Identify Gestures


41


1. Description

Use case allows machine to recognize gestures.

2. Actor

Machine

3. Flow of event

3.1Basic flow-




3.1.4 Detect Hand.

3.1.4 Identify gesture.



5. Precondition

5.1 Camera is on


6. Post condition

6.1 Gesture identified successfully.

3.2.4 Use case specification: Perform Operations


42


1. Description

Use case allows machine to perform operations depending on gestures.

2. Actor

Machine

3. Flow of event

3.1Basic flow-




3.1.4 Detect Hand.

3.1.4 Identify gesture.

3.1.5 Perform operation depending on gesture.



5. Precondition

5.1 Camera is on


6. Post condition

6.1 Operation performed successfully depending on gesture made.

3.3 Use case Analysis:


43


3.3.1 Activity Diagram:

User Machine

Switch on cam Cam switched on

Moves hand to cam Detects Wrist Band

Detect region of Interest

Makes gestures Captures image

processes image

Identify gesture

Perform Operation Report error

[gesture identified] [gesture not identified]

Detect hand

Fig.3.3.1 Activity Diagram

3.3.2 Sequence Diagram:


44


ALtsd

User Machine

1 : power on cam()

2 : Cam on()

3 : Moves hand to cam()

4 : Detect Wrist Band()

5 : Detect Region Of Interest()

6 : Detect Hand()

7 : Make gestures()

8 : Capture image()

9 : Process image()

10 : Identify Gesture()

11 : Perform operation()

12 : Error()

13 : Displays error message()

Fig.3.3.2 Sequence Diagram

CHAPTER 4


45


PROJECT DESIGN

4.1 Design Model:

4.1.1 Class Diagram:

Machine

+int cam+int fun+int temp

+int check()+int WBdetect(int color)+int hdetect()+void improcess()+pform_operation()

User

+char name+int hand+int temp1

+int cam()+void makegesture()

gives input to

Fig.4.1.1 Class Diagram

4.1.2Data Flow Diagrams:SCOE, Information Technology, 2010-11.

46


DFD Level 0:

Fig.4.1.2a DFD Level 0

DFD Level 1:


47


Fig.4.1.2b DFD Level 1

4.2 Implementation Model:


48


4.2.1 Component Diagram:

Computer

Webcam

Users

captured gestures

Hand movements

Fig.4.2.1 Component Diagram

CHAPTER 5


49


TESTING

5.1Test case Design- Unit/Integration:

Manual Testing:-

Manual testing is the process of manually testing software for defects. It requires

a tester to play the role of an end user, and use most of all features of the application to

ensure correct behavior. To ensure completeness of testing, the tester often follows a

written test plan that leads them through a set of important test cases. It is the oldest and

most rigorous type of software testing. Manual testing requires a tester to perform manual

test operations on the test software without the help of Test automation. Manual testing is

a laborious activity that requires the tester to possess a certain set of qualities; to be

patient, observant, speculative, creative, innovative, open-minded, resourceful,

unopinionated, and skillful.

Unit Testing:-

This initial stage in testing normally carried out by the developer who wrote the

code and sometimes by a peer using the white box testing technique. The most 'micro'

scale of testing; to test particular functions or code modules. Typically it is done by the

programmer and not by testers, as it requires detailed knowledge of the internal program

design and code. Not always easily done unless the application has a well-designed

architecture with tight code; may require developing test driver modules or test harnesses.


50

http://en.wikipedia.org/wiki/Test_case

http://en.wikipedia.org/wiki/Test_plan


Here in our project we have tested each module uniquely by using unit testing.

We have provided facility to recover project when error occurs.

Integration testing:-

It is a testing of combined parts of an application to determine if they function

together correctly. The 'parts' can be code modules, individual applications, client and

server applications on a network, etc. This type of testing is especially relevant to

client/server and distributed systems. This stage is carried out in two modes? viz?. As a

complete package or as a increment to the earlier package. Most of the time black box

testing technique is used. However, sometimes a combination of Black and White box

testing is also used in this stage.


51


Test cases:

Table 5.1.1 Wristband detection test case

Project: Controlling Computer Using Hand Gesture Recognition.

Module: Wrist band detectionForm REF: MainTest Case No: 1Functional Specification

Wrist band detection

Test Date: 9/04/11 1minTest Objective: To check whether user has put

wrist band on or not.Test Data: Wrist bandPrecondition Camera should be onStep No Steps Data Expected Results Actual Results

1 Use hand without wrist band. Camera input Show message as put wrist band on

YES, message is displayed

2 Use wrist band with color other than yellow.

Camera input Show message as put wrist band on, as wrist band other than yellow color is not detected.

YES, message is displayed

3 Use wrist band with yellow color.

Camera input Wrist band is detected and ROI of wrist band is calculated.

YES, wrist band is detected properly.

Table 5.1.2 Hand segmentation test case


Module: Hand segmentation


52


Form REF: MainTest Case No: 2Functional Specification

Segmentation of hand

Test Date: 9/04/11 1minTest Objective: To check whether hand is

segmented properly or not.Test Data: Hand with wrist bandPrecondition Camera should be on.Step No Steps Data Expected Results Actual Results

1 Use hand without wrist band in front of camera.

Camera input Show message as put wrist band on.

YES, message is displayed and hand is not segmented.

2 Use hand with wrist band with color other than yellow in front of camera.

Camera input Show message as put wrist band on, as wrist band other than yellow color is not detected.

YES, message is displayed and hand is not segmented.

3 Use hand with wrist band of yellow color in front of camera..

Camera input Wrist band is detected and ROI of hand is calculated and hand is segmented.

YES, wrist band is detected and hand is segmented properly.

Table 5.1.3 Gesture identification test case


Module: Gesture identificationForm REF: MainTest Case No: 3Functional Specification

Identification of hand

Test Date: 9/04/11 1minTest Objective: To check whether gesture is

identified properly or not.Test Data: Hand gesturesPrecondition 1) Camera should be

on.


53


2) Hand with yellow wrist band.

Step No Steps Data Expected Results Actual Results1 Hand with unspecified gestures. Camera input Wait for specified

gestures.Do nothing.

2 Hand with specified gestures. Camera input Gesture is identified properly.

YES, gesture is identified and message is displayed(e.g. left click, right click etc.)

Table 5.1.4 Perform operations test case


Module: Perform operationsForm REF: MainTest Case No: 4Functional Specification

Implementations of operations

Test Date: 9/04/11 1minTest Objective: To check whether operations are

performed correctly or not.Test Data: Hand gesturesPrecondition 1) Camera should be

on.2) Hand with yellow

wrist band.Step No Steps Data Expected Results Actual Results

1 Hand with unspecified gestures. Camera input Wait for specified gestures and no operations performed.

Do nothing.


54


2 Hand with specified gestures. Camera input Operations are performed depending on gestures.

YES, operations are performed properly.

Table 5.1.5 Left click test case



Implementations of left click

Test Date: 9/04/11 1minTest Objective: To check whether left click is





Do nothing.

2 Hand with only index finger and thumb are straight and angle between them is acute.

Camera input Left click operation is performed.

YES, left click is performed properly.


55


Table 5.1.6 Cursor movement test case



Implementations of cursor movement

Test Date: 9/04/11 1minTest Objective: To check whether cursor

movement operation performed correctly or not.

Test Data: Hand gesturesPrecondition 1) Camera should be




Do nothing.

2 Hand with only index finger is straight and all other fingers are curled and movement of hand takes place.

Camera input Cursor moves according to hand movements.

YES, cursor movement takes place.


56


Table 5.1.7 Right click test case



Implementations of right click

Test Date: 9/04/11 1minTest Objective: To check whether right click is





Do nothing.

2 Hand with only index finger and middle fingers are straight and all other fingers are curled.

Camera input Right click operation is performed.

YES, right click operation is performed.

CHAPTER 6

CONCLUSIONS


57


The goal of this project is to create a system that will recognize the hand gestures

and control Computer according to those gestures. The project will benefit the mobile

systems where using pointing devices like mouse is difficult. In this project video camera

captures 25 fps. It is considered that a modern computer system would therefore allow

the project goals to be exceeded.

Before actual implementing operations, wrist band detection, determining region

of interest, hand segmentation and identifying gestures from stored frames need to be

done. After detecting wristband, band to hand ratio is calculated, which is helpful in

segmenting hand from background image. Convex hull and Convex defect techniques are

used to find actual hand image and depending on that gestures identified. After gesture

identification we need to implement those gestures to system level mouse and keyboard

instructions.

In future project can be used for multimedia applications like playing games

without using mouse also all keyboard operations can be performed using this project.

But enhancing this project is quite complex and difficult, so lots of hard work is required.

However, it is difficult to get stable results because of the variety of lighting and

skin colors of human races. Most vision algorithms have illumination issues. From the

results, we can expect that if the vision algorithms can work in all environments then our

system will work more efficiently. This system could be useful in presentations and to

reduce work space.

CHAPTER 7

BIBLIOGRAPHY


58


[1].[Bauer & Hienz, 2000] Relevant feature for video-based continuous sign language

recognition. Department of Technical Computer Science, Aachen University of

Technology, Aachen, Germany, 2000.

[2].[Davis & Shah, 1994] Visual gesture recognition. In proceedings IEEE Visual

Image Signal Process, 1994, vol.141, No.2, pages 101-106.

[3].J. Brand and J. Mason, "A Comparative Assessment of Three Approaches to

Pixel-level Human Skin Detection", IEEE International Conference on Pattern

Recognition, 2000.

[4].Mark Tabb and Narendra Ahuja, “Multiscale Image Segmentation by Integrated

Edge and Region Detection”, IEEE Transactions on Image Processing, vol. 6, no.

5, May 1997.

[5].R. C. Gonzalez and R. E. Woods, Digital Image Processing, Prentice-Hall, 2nd

edition, 2002.

[6].Zhi Li, Ray Jarvis, “Real time Hand Gesture Recognition using a Range Camera”.


59

Final Report 21-4

Documents

Transcript of Final Report 21-4