Complete 8th Sem Report
Transcript of Complete 8th Sem Report
i
APPARATUS AND METHOD FOR GESTURE
RECOGNITION
A PROJECT REPORT
Submitted by
PRATIK BHATT (080110111003) DEVAL MEHTA (080110111021)
NIRAV FARASWAMI (080110111012)
In fulfillment for the award of the degree
of
BACHELOR OF ENGINEERING
In
Electronics and Communication
G.H.PATEL COLLEGE OF ENGINEERING AND TECHNOLOGY, V.V.NAGAR
Gujarat Technological University, Ahmedabad
MAY, 2012
ii
CERTIFICATE
Date: 23-MAY-2012.
This is to certify that the dissertation entitled “APPARATUS AND METHOD FOR
GESTURE RECOGNITION ” has been carried out by PRATIK BHATT,
DEVAL MEHTA and NIRAV FARASWAMI under my guidance in
fulfillment of the degree of Bachelor of Engineering in Electronics and
Communication Engineering (8th Semester) of Gujarat Technological
University, Ahmedabad during the academic year 2011-12.
Project Guide: Head of Department
Prof. NILESH H DESAI. Dr. Chintan .K. Modi
G.H.PATEL COLLEGE OF ENGINEERING AND TECHNOLOGY
ELECTRONICS AND COMMUNICATION
2012
iii
ACKNOWLEDGEMENT
We are extremely thankful to our project guide Prof. Nilesh H Desai sir for being a
constant source of inspiration and encouragement. We would like to express our gratitude
towards him for solving our queries and helping us for the project.We are thankful to our
friends and families for their support and assistance.
iv
Abstract
In this world where every operation performed by human are getting simple, in order to make it more simpler we present an innovation which makes use of human GESTURE ,the very definition of being human. The invention can make dumb people speak, make disables people operate wheelchair, and can facilitate the life of human by providing HOME-AUTOMATION program e.g. controlling your Televisions and Air-conditioners just by making appropriate gesture.
The biggest advantage of this invention is that it is cost effective since image-processing is not used, thus reducing hardware and hence the cost. Device is made up of basic electronic components like IR sensor (IR LED and Photo-diode) in order to recognize gesture, and other basic processing unit such as microcontroller. The sensors are basically a proximity sensors which converts position of Fingers either open or close into proper electrical signal or logical level like “0” and “1”.
This data is then processed by microcontroller unit to generate different codes or signals for remote applications. Also for the application of speaking aid to dumb people with further use of Text to speech converters we can convert gesture into sound and hence can make dumb people speak.
v
List of figures
Figure Number
Figure Description Page no.
Fig 3.1 Finger is open 6 Fig 3.2 Finger is closed 6 Fig 3.3 Gesture for alphabet A with Code: 10000 7 Fig 3.4 Gesture for alphabet B with Code: 11110 7 Fig 3.5 Gesture for alphabet C with Code: 1111111 7 Fig 3.6 Gesture for alphabet E with Code: 111 8 Fig 3.7 Gesture for alphabet F with Code: 100011 8 Fig 3.8 Gesture for alphabet G with Code: 00001 8 Fig 3.9 Gesture for alphabet H with Code: 101001 8 Fig 3.10 Gesture for alphabet I with Code: 1000 9 Fig 3.11 Gesture for alphabet J with Code: 1000001000 9 Fig 3.12 Kinect gesture recognizing device 10 Fig 3.13 eyeSight’s gesture recognizing device 10 Fig 3.14 AcceleGlove’s gesture recognizing 11 Fig 4.1 General Block Diagram 14 Fig 4.2 General LED Construction 16 Fig 4.3 Surface mount LED 16
vi
Fig 4.4 Fig 4.5 Fig 4.6 Fig 4.7 Fig 4.8 Fig 4.9 Fig 4.10 Fig 4.11 Fig 4.12 Fig 5.1 Fig 5.2 Fig 5.3 Fig 5.4 Fig 5.5 Fig 5.6 Fig 5.7 Fig 5.8 Fig 5.9 Fig 5.10 Fig 6.1 Fig 6.2 Fig 6.3 Fig 6.4 Fig 6.5
Characteristics of Photodiode 89c2051 Interfacing figure NE555 Interfacing figure ST3654 Interfacing figure TTS256 DIP Speakjet DIP TTS256 and Speakjet Interfacing figure Character LCD HT6230 DIP Right hand Microcontroller unit along with sensors Left hand Microcontroller unit along with sensors Neck Microcontroller unit along with TTS256 and Speakjet Finger open condition detailing Finger closed condition detailing Gesture operated T.V. Front view of sensors Front view of left hand with touchpad Front view of right hand with touchpad Position of touchpad for thumb Proteus simulation Right hand module Left hand module Completed module Breadboard implementation of TTS256 module
17 21 21
23 25 26 27 28 30 31 31 32 32 33 33 34 36 36 37 41 42 43 44 45
vii
LIST OF ABBEREVATIONS
SYMBOL NAME ABBREVIATIONS
IR Infra Red
R Red
LED Light Emitting Diode
viii
LIST OF TABLES
Table no. Table Description Page no.
4.1 Comparison of Different MCU 19
4.2 TTS256 Pin Diagram 25
4.3 LCD Pin Diagram 28
ix
Table of contents
Acknowledgement iii
Abstract iv
List of figures v
List of abbreviations vii
List of Tables viii
Table of contents ix
Chapter 1: Introduction 1
Chapter 2: Brief history of Gesture Recognizing methods 2
Chapter 3: Literature review 5
3.1 Gesture Recognition 5
3.1.1 Basics of Gesture Recognition 5
3.1.2 Principles of operation of Gesture Recognition Technology 6
3.1.3 Alphabetic Gestures and corresponding binary codes 7
3.2 Availability of system in market 9
3.2.1 Kinect for Windows, Gesture recognizing UI 9
3.2.2 eyesight’s Hand Gesture Recognizing Technology 10
3.2.3 AcceleGlove 11
3.3 Applications 12
3.4 Motivation and objectives 13
Chapter 4 : Gesture Recognition system and its components 14
x
4.1 Block Diagram 14
4.2 Sensors 15
4.2.1 IR LED 15
4.2.2 Photodiode 17
4.3 Left hand microcontroller unit 18
4.3.1 89c2051 18
4.3.2 NE555 Timer IC 21
4.3.3 RF Module 22
4.3.3.1 ST3654- Serial Interface IC 22
4.3.3.2 RF Transreciever 23
4.3.4 Right hand Microcontroller unit 24
4.3.5 TTS256 24
4.3.6 Speakjet IC 25
4.3.7 Character LCD 28
4.3.8 HT6230 IC 29
Chapter 5: Working/Implementation of Project Work 31
5.1 Drawings of method using IR 31
5.2 Working of the model using IR 34
5.3 Drawings of the method using touchpad 36
5.4 Working of the model using touchpad 38
5.5 Algorithm of C-program 39
Chapter 6: Result Analysis 41
6.1 Proteus Simulation 41
6.2 Hardware Results 42
Chapter 7: Conclusion 46
References 47
Rewards 48
1
Technological Revolution has given us a powerful tool to control our surrounding.
We recognize this tool as Remote controls. Today everything can be operated using
Remote controls, from T.Vs to door locks. Want to change the channel? Just click the
button. But in this process we have lost something much more valuable i.e. Being
Gesture oriented. Gestures are an important means of human being to express their
thoughts in the best possible way. So we came up with the questions: why can’t we
operate things around us just by our hand gestures? Why can’t we just change the
channels or switch on the lights with some gestures?
After detail research on gesture recognition, we were certain about one thing i.e. gesture
recognition need a CAMERA and a digital Processor (mainly laptops). The idea of
carrying a laptop on the back and camera on the head did not appeal to us that much. So
we challenged this notion and started developing an innovative concept for gesture
recognition which must use only basic electronics components and no cameras. The end
result is what we call now “Gesture speaks”.
“Gesture speaks” is a device which once recognizes gesture made by hand fingers.
The device has to be worn on the hands. The sensors consist of rings which are to be
worn on each finger. The sensors are basically proximity sensors which are discussed in
more detail later. They constantly monitor the position of the finger i.e. whether they
are open or close. On the basis of data of position of all the fingers, the device
decides which gesture is invoked.
INTRODUCTION Chapter-1
2
Gesture recognition is a topic in computer science and language technology with the
goal of interpreting human gestures via mathematical algorithms. Gestures can originate
from any bodily motion or state but commonly originate from the face or hand. Current
focuses in the field include emotion recognition from the face and hand gesture
recognition. Many approaches have been made using cameras and computer vision
algorithms to interpret sign language. However, the identification and recognition of and
human behaviors are also the subject of gesture recognition techniques.
Gesture recognition enables humans to interface with the machine (HMI) and
interact naturally without any mechanical devices. Using the concept of gesture
recognition, it is possible to point a finger at the computer screen so that the cursor will
move accordingly. This could potentially make conventional input devices such as
mouse, keyboards and even touch-screens redundant. Gesture recognition has been being
conducted with techniques from computer vision and image processing, up till now.
The literature includes work done on gesture recognition without using image
processing and computer vision, without using any kind of memory for storage, just
using very basic sensors of IR transmitter and receiver pair.
In computer interfaces, two types of gestures are distinguished:
�Offline gestures: Those gestures that are processed after the user interaction with the
object. An example is the gesture to activate a menu.
�Online gestures: Direct manipulation gestures. They are used to scale or rotate
�For socially assistive robotics: By using proper sensors (accelerometers and gyros)
worn on the body of a patient and by reading the values from those sensors, robots
can assist in patient rehabilitation. The best example can be stroke rehabilitation.
�Directional indication through pointing: Pointing has a very specific purpose in our
society, to reference an object or location based on its position relative to ourselves. The
Brief history of Gesture Chapter-2
Recognizing methods
3
use of gesture recognition to determine where a person is pointing is useful for
identifying the context of statements or instructions. This application is of particular
interest in the field of robotics.
�Control through facial gestures: Controlling a computer through facial gestures is a
useful application of gesture recognition for users who may not physically be able to use
a mouse or keyboard. Eye tracking in particular may be of use for controlling cursor
motion or focusing on elements of a display.
�Alternative computer interfaces: Foregoing the traditional keyboard and mouse setup
to interact with a computer, strong gesture recognition could allow users to
accomplish frequent or common tasks using hand or face gestures to a camera.
�Immersive game technology: Gestures can be used to control interactions within
video games to try and make the game player's experience more interactive or
immersive.
�Virtual controllers: For systems where the act of finding or acquiring a physical
controller could require too much time, gestures can be used as an alternative control
mechanism. Controlling secondary devices in car or controlling a television set are
examples of such usage.
�Affective computing: In affective computing, gesture recognition is used in the
process of identifying emotional expression through computer systems.
�Remote control: Through the use of gesture recognition, "remote control with the
wave of a hand" of various devices is possible. The signal must not only indicate the
desired response, but also which device to be controlled.
Input devices used up till now in this discipline as followed:
�Wired gloves: These can provide input to the computer about the position and rotation
of the hands using magnetic or inertial tracking devices. Furthermore, some gloves
can detect finger bending with a high degree of accuracy (5-10 degrees), or even provide
haptic feedback to the user, which is a simulation of the sense of touch. The first
commercially available hand-tracking glove-type device was the Data Glove, a glove-
type device which could detect hand position, movement and finger bending. This uses
fiber optic cables running down the back of the hand. Light pulses are created and
when the fingers are bent, light leaks through small cracks and the loss is registered,
giving an approximation of the hand pose.
4
�Depth-aware cameras: Using specialized cameras such as time-of-flight cameras, one
can generate a depth map of what is being seen through the camera at a short range, and
use this data to approximate a 3d representation of what is being seen. These can be
effective for detection of hand gestures due to their short range capabilities.
�Stereo cameras: Using two cameras whose relations to one another are known, a 3d
representation can be approximated by the output of the cameras. To get the cameras'
relations, one can use a positioning reference such as a lexian-stripe or infrared
emitters. In combination with direct motion measurement (6D-Vision) gestures can
directly be detected.
�Controller-based gestures: These controllers act as an extension of the body so that
when gestures are performed, some of their motion can be conveniently captured by
software. Mouse gestures are one such example, where the motion of the mouse is
correlated to a symbol being drawn by a person's hand, as is the Wii Remote, which can
study changes in acceleration over time to represent gestures. Devices such as the LG
Electronics Magic Wand, the Loop and the Scoop use Hillcrest Labs' Freespace
technology, which uses MEMS accelerometers, gyroscopes and other sensors to translate
gestures into cursor movement. The software also compensates for human tremor and
inadvertent movement.
�Single camera: A normal camera can be used for gesture recognition where the
resources/environment would not be convenient for other forms of image-based
recognition. Although not necessarily as effective as stereo or depth aware cameras,
using a single camera allows a greater possibility of accessibility to a wider audience.
5
3.1 Gesture recognition
3.1.1 Basics of gesture recognition:
With the use of sensor, fabricated with IR LED and Photodiode, the position of the
fingers is detected for both the hands. The sensors are to be worn on both hands, on each
individual finger. The sensors are in the form of a ring. All these rings are connected to
Microcontroller and Interfacing unit, where continuous polling of the fingers is done,
whether it is open or close. This procedure is repeated periodically.
Corresponding to the results of the polling of each finger, rather the position of the finger
being detected, a unique code for each position of the fingers is generated. The code
generated is different for each alphabet (A-Z) and numbers from 0 to 9. The position of
the fingers and the corresponding codes are designed according to American
standards for sign language of Deaf and Dumb people. There is code assigned for
SPACE between the words in a sentence and all for any kind of erroneous gesture
generated by the user, other than the predefined position of the fingers.
Care has been taken, considering the normal human tendencies for movement of the
hands, especially taking into account the thumb finger and tiny finger, for each gesture.
The polled data of fingers of the left hand is send via RF transmission using RF module,
to the right hand RF module, which is further processed in the right hand
Microcontroller unit.
Taking into consideration of the position of fingers of both the hands, a unique code
is generated from right hand Microcontroller unit and it is again send serially, via RF
communication to the RF detector module worn on the neck. The code is further
converted into ANCII letters and numbers using microcontroller. IC TTS256 converts
ANCII letters into allophonic codes. This allophone IC converts the allophonic codes
into corresponding electrical signals and it is further send to speaker.
LI TERATURE SURVEY Chapter-3
6
3.1.2 Principles of operation of gesture recognition:
Fig 3.1 Finger is open
Initially when the finger is open, the IR LED throws its ray in the direction as shown in
the figure. This ray is not sensed by the photodiode, thus resulting in no change in
the resistance of the photodiode and no change in the voltage level.
Fig 3.2 Finger is closed
7
When the finger is closed, the ray from the IR is detected by the photodiode. This
further changes the resistance of the photodiode and correspondingly changes the
voltage level. This change of voltage level is polled by the microcontroller and it is
further processed.
3.1.3 Alphabetic gestures and corresponding binary codes
Fig 3.3 Gesture for alphabet A with Code:10000
Fig 3.4 Gesture for alphabet B with Code: 11110
Fig 3.5 Gesture for alphabet C with Code: 1111111
8
Fig 3.6 Gesture for alphabet D with Code: 111
Fig 3.7 Gesture for alphabet E with Code: 100011
Fig 3.8 Gesture for alphabet G with Code: 00001
Fig 3.9 Gesture for alphabet H with Code: 101001
9
Fig 3.10 Gesture for alphabet I with Code: 10000
Fig 3.11 Gesture for alphabet J with Code: 1000001000
3.2 Availability of systems in market
There are many models and brands of gesture recognizing systems in different fields.
They are described with their brief features as followed.
3.2.1 Kinect for Windows, Gesture-recognizing UI around the Corner
After Microsoft evolved game controller the Kinect for Xbox 360 came to the fore,
speculation among tech circles saw a future for the device in the PC platform right
away, beyond being just a game controller. It looks like Redmond is taking steps in that
direction, with the groundwork for Kinect's arrival on the Windows PC platform
underway. Microsoft will release the Kinect for Windows software development kit
(SDK) this spring, so developers can start work on it right away. This could include
giving Games for Windows applications Kinect support which they enjoy on the
Xbox 360 platform.
10
Fig 3.12 Kinect gesture recognizing device
The possibilities are endless for non-game applications to make use of Kinect as a
gesture-recognizing and face-recognizing device. Gesture recognition UI and face-
recognition are pitched to be some of the defining features of Microsoft's next version of
Windows. Kinect for Windows SDK will be released on a non-commercial basis. "The
hope is that the SDK will unleash a wave of creativity to add to the already
exciting developments we’ve seen on top of Kinect. The SDK will provide access to
Kinect’s sensor as well audio and system API’s," the company commented.
3.2.2 eyeSight’s Hand Gesture Recognition Technology
Allows people to interact with devices using simple hand gestures. eyeSight’s solution
tracks the user’s hand motions in front of the device’s camera and converts these
gestures into user input commands that control the device and its applications. eyeSight’s
solution is based on advanced image processing and machine vision algorithms, which
analyze real time video input from common built-in cameras. The technology is
independent of the underlying processor and camera hardware and produces high
quality gesture recognitions using low-end VGA cameras.
Fig 3.13 eyeSight’s gesture recognizing device
11
Moreover, the technology is designed for embedded platforms. It is optimized to operate
utilizing minimal CPU and power consumption and supports challenging user
environments with poor as well as direct lighting conditions. eyeSight’s Hand Gesture
Recognition Technology is available for integration at any layer, either as part of the
device software stack or as low as the camera sensor silicon level.
3.2.3 AcceleGlove
The AcceleGlove is an instrumented glove that can be used as a
communication and control device for recognizing hand gestures or controlling
computers and robotic devices.
Fig 3.14 AcceleGlove’s gesture recognizing
The AcceleGlove contains six accelerometers on the fingers and the back of the hand
that sense hand and finger orientation and movements. Pattern recognition software
captures and recognizes static hand positions, as well as dynamic gestures.
What Will It Accomplish?
� Provides an ideal interface for recognizing gesture-based communication such as sign
language or military hand signals
�Provides an intuitive interface for computers, gaming devices, virtual environments
and visual displays
�Provides an intuitive interface for controlling robotic devices such as unmanned
vehicles and payloads such as robotic arms or cameras
�Provides a means for tracking hand movements for medical assessments and
rehabilitation.
12
3.3 Applications
1. The user can operate any home appliances like T.V, A.C or Microwave just by
making gestures. For e.g. if you want to change the channel to 24? Just point “2” with
two fingers and “4” with four fingers and the channel will be changed. Want to increase
the volume? Just point your thumbs up till the volume is needed to be changed. Same is
the case to change temperature in A.Cs and Microwave. Thus we don’t need to carry
different remotes for different appliances. Just by selecting different modes in this device
you can switch from operating A.C to T.V. Thus anything that can be controlled with
remote controls can be controlled with our device.
2. For disabled people using wheelchair, the user can operate the wheelchair with
his/her gestures. Want to go straight? Point your index finger. Want to turn left? Point
your right thumb. Want to turn right? Point your left thumb. Thus it can help the user the
better operate the wheel chair rather than manual effort or use of joysticks. The same
technique is applicable for controlling a ROBOT with gestures.
3. The most striking feature of this device is the ability of converting “Gesture into
speech”. Dumb people with speaking problems are resorted to sign language. But it is
very difficult to convey their message to a person not familiar with the sign language.
This innovation will give them voice and they will be able to convey their message
via speech rather than signs. For e.g. A dumb person just has to sign “C” “A” “T” and
device will speak “CAT”. Thus it is like adding voice to their sign language and it
will be nothing less than a boon to them.
4. For kinder-garden kids and small children to learn English alphabets and Number
system with gestures e.g. “Gesture speaks” understands sign language for every
English alphabet and numbers. Gesturing “1” or ‘4” will speak “Fourteen”. Gesturing
“A” will speak “A for Apple”.
5. Also the device can work as a Gesture controlled calculator. Gesture “1”, “+”
with two index fingers, then “5” and “=” and the device will either display or speak the
answer “six”. Thus it can save the time of pressing buttons repeatedly.
6. The above listed applications are just small part of what can be done with this device.
Applications are only limited to human imagination. Future application can vary from
controlling your laptop to change songs or ppt slides to gesture controlled mouse.
Another striking application is Gesture mobile. Interfacing GSM module with this device
13
can call or SMS any number without the need of actual mobile. Just gesture the number
and gesture “dial” and the call will be connected.
3.4 Motivation and Objectives
The present invention is in the technical field of electronics. More particularly the
present invention is in the technical field of optical gesture recognition through use
of simple sensors like IR LED and IR photodiode. Gesture recognition is mainly done
through image processing in present days but difficulty with this technique is the
complexity of hardware and software required. Also the cost related to such
technique is high, thus it cannot be used for home applications very efficiently.
This device can control different homemade appliances like Televisions, Air- conditions,
compact disk player. Further assigning every English alphabet a unique gesture, it can
convert “gestures into speech” and thus can be used as a speaking aid for dumb
people. This invention allows the deaf people to communicate with the normal
human beings in the world easily, with the ease for the people to easily understand their
sign language which is converted into speech. Also this device can be used for gesture
controlled wheelchair for disabled people.
14
4.1 Block Diagram:
Fig 4.1 General Block Diagram
List of system components are as follows:
1 Sensors
2 Left hand microcontroller unit
3 RF modem
4 Right hand microcontroller unit
APPARATUS FOR Chapter 4
GESTURE RECOGNITION
AND IT’S COMPONENTS.
15
5 TTS256 IC
6 Speakjet IC
7 Graphics LCD.
8 HT6230.
4.2 Sensors
4.2.1 IR LED
The LED consists of a chip of semiconducting material doped with impurities to create a p-n junction. As in other diodes, current flows easily from the p-side, or anode, to the n-side, or cathode, but not in the reverse direction. Charge-carriers— electrons and holes—flow into the junction from electrodes with different voltages. When an electron meets a hole, it falls into a lower energy level, and releases energy in the form of a photon.
The wavelength of the light emitted, and thus its color depends on the band gap energy of the materials forming the p-n junction. In silicon or germanium diodes, the electrons and holes recombine by a non-radiative transition, which produces no optical emission, because these are indirect band gap materials. The materials used for the LED have a direct band gap with energies corresponding to near-infrared, visible, or near-ultraviolet light.
16
Fig 4.2 General LED Construction
LED development began with infrared and red devices made with gallium arsenide. Advances in materials science have enabled making devices with ever- shorter wavelengths, emitting light in a variety of colors.
LEDs are usually built on an n-type substrate, with an electrode attached to the p-type
layer deposited on its surface. P-type substrates, while less common, occur as well. Many
commercial LEDs, especially GaN/InGaN, also use sapphire substrate.
Most materials used for LED production have very high refractive indices. This
means that much light will be reflected back into the material at the material/air surface
interface. Thus, light extraction in LEDs is an important aspect of LED
production, subject to much research and development.
Fig 4.3 Surface mount LED
17
4.2.2 Photodiode
A photodiode is a p-n junction or PIN structure. When a photon of sufficient energy
strikes the diode, it excites an electron, thereby creating a free electron (and a positively
charged electron hole). This mechanism is also known as the inner photoelectric effect.
If the absorption occurs in the junction's depletion region, or one diffusion length away
from it, these carriers are swept from the junction by the built- in field of the depletion
region. Thus holes move toward the anode, and electrons toward the cathode, and a
photocurrent is produced. This photocurrent is the sum of both the dark current (without
light) and the light current, so the dark current must be minimized to enhance the
sensitivity of the device.
Fig 4.4 Characteristics of Photodiode
�Critical performance parameters of a photodiode include:
Responsivity-
The ratio of generated photocurrent to incident light power, typically expressed in
A/W when used in photoconductive mode. The responsivity may also be expressed as
Quantum efficiency, or the ratio of the number of photogenerated carriers to incident
photons and thus a unit less quantity.
Dark current-
The current through the photodiode in the absence of light, when it is operated in
18
photoconductive mode. The dark current includes photocurrent generated by background
radiation and the saturation current of the semiconductor junction. Dark current must be
accounted for by calibration if a photodiode is used to make an accurate optical power
measurement, and it is also a source of noise when a photodiode is used in an optical
communication system.
4.3 Left hand microcontroller unit
4.3.1 89c2051
Low-voltage, high-performance CMOS 8-bit microcomputer with 2KB of flash
programmable and erasable read-only memory. The device is manufactured using
Atmel high-density nonvolatile memory technology and is compatible with the industry-
standard MCS-51 instruction set. This versatile 8-bit CPU with flash provides a
highly flexible and cost-effective solution for many embedded control applications.
Features:
• Compatible with MCS®-51Products
• 2K Bytes of Reprogrammable Flash Memory
– Endurance: 1,000 Write/Erase Cycles
• 2.7V to 6V Operating Range
• Fully Static Operation: 0 Hz to 24 MHz
• Two-level Program Memory Lock
• 128 x 8-bit Internal RAM
• 15 Programmable I/O Lines
• Two 16-bit Timer/Counters
• Six Interrupt Sources
19
• Programmable Serial UART Channel
• Direct LED Drive Outputs
• On-chip Analog Comparator
• Low-power Idle and Power-down Modes
• Green (Pb/Halide-free) Packaging Option
Role of Microcontroller Unit:
The Microcontroller unit is used for continuous polling of the fingers, whether they are
closed or open. Depending on this result, it generates a unique code which is send via RF
module to the right hand controller unit.
Selection of Microcontroller:
Various type of microcontroller having different kind of features is available in market
such as 8051, PIC, AVR, ARM, TI-MSP430. Various parameters lie power consumption,
cost, availability of ADC/DAC, UART, Timers, processing speed etc. determines the
choice of microcontroller. The choice of microcontroller must be in accordance with
requirements of project.
Table 4.1 Comparison of different MCU
Features 89c2051 PIC16F87
XA
AVR MSP430F26
18
MSP430FG43X
Supply
Voltage
5 V 5 V 4.5-5.5
V
1.8V to 3.6V 1.8V to 3.6V
Frequency 0-24MHz DC-20
MHz
0-16
MHz
0-16MHz 0-8 MHz
RAM 128Bytes 368Bytes 1KB 4KB 2KB
20
ROM 2KB 128Bytes
EEPROM
512byt
es
EEP
120KB +
256B
Flash
60KB+256B
Flash Memory
Input/outpu
t lines
15 22 32 64 48
Architectur CISC RISC RISC RISC RISC
Instruction
set
111 35 131 51 51
Timer Two 16-bit Three
timers-
Timer0-8-
bit,
Timer1-
16-bit,
Timer2-8-
bit
Three
flexib
le
timers
with
comp
are
mode
16-bit timer
A
with three
capture/com
pare
registers.
16-bit timer
B with
16-bit timer A
with three
capture/compare
registers.
16-bit timer B with
seven compare with
shadow registers
ADC NA 10 bit ADC 10-bit
ADC
12-bit ADC 12-bit ADC
DAC NA NA NA Dual 12-bit
DAC
Dual 12-bit DAC
with synchronization
Inbuilt
OPAMP
NA NA NA NA 3 inbuilt
configurable
opamps.
Power
Down
Two
power
saving.
Brown out
reset
Six
softw
are
power
savin
5 low power
modes
5 power saving
modes.
Serial
interface
Programm
abl
e full
duplex
serial
channel.
USART Progra
m
mabl
e
Serial
USA
Four USCI USART ,
Asynchronous
UART
21
Justifying the choice of 89c2051:
The project requirement for left hand side microcontroller clearly justifies the use of only
5 ports for polling of fingers, only one timer and use of full duplex serial channel.
Considering the cost requirement and choice of other microcontroller, 89c2051
justifies its presence in the hardware.
The interfacing figure used in the circuit for 89c2051 is as followed:
Fig 4.5 89c2051 Interfacing figure
4.3.2 NE555 Timer IC
The timer IC is used for generating 38 KHz square wave for providing the pulse to
the IR LED so that it can emit infrared light. The interfacing figure of this IC for
generating 38 KHz pulse is shown below.
22
Fig 4.6 NE555 Interfacing figure
4.3.3 RF module
The RF module is used for quick serial transmission and reception operating it into
USART mode. This module justifies a secured serial transmission for a short range
of 30 meters, which is requirement of the project.
4.3.3.1 ST3654 – Serial Interface IC
ST3654 Serial Interface provides a simple UART interface for transmission and reception
of serial data at various baud rates. It can be used for applications that need two way
wireless data transmission. The communication protocol is self-controlled and
completely transparent to user interface. The IC can be embedded to your current design
so that wireless communication can be set up easily.
Features:
�Automatic switching between TX and RX mode with LED indication
�Adjustable baud rate setting of 9600, 4800, 38400 and 19200
�FSK technology, half duplex mode, robust interference
�Protocol translation is self-controlled, easy to use
23
�High sensitivity, optimized transmission range.
�Standard UART interface, TTL(3-5V) logic level with any microcontroller
�Very reliable, small size, easier mounting
�No tuning required, PLL based self-tuned
�Error checking(CRC) to prevent corrupted data output at receiver
The interfacing figure of the ST3654 with RF transreciever is as shown below.
Fig 4.7 ST3654 Interfacing figure
4.3.3.2 RF Transreceiver 2.4 Ghz
This is an FSK Transceiver module is a true single-chip transceiver, it is based on 3
wire digital serial interface and an entire Phase-Locked Loop (PLL) for precise local
oscillator generation. It can use in UART/ NRZ / Manchester encoding / decoding.
It is a high performance and low cost module. It gives 30 meters range with onboard
antenna. Operating Range is 30 meters without requiring any external antenna.
24
Features:
�Low power consumption.
�Integrated bit synchronizer.
�Integrated IF and data filters.
�High sensitivity (type -104dBm)
�Programmable output power -20dBm~1dBm
�Operation temperature range : -40~+85 deg C
�Operation voltage: 1.8~3.6 Volts.
�Available frequency at : 2.4~2.483 GHz
4.3.4 Right hand microcontroller unit
This block has same specifications and the corresponding interfacing unit as the left
hand controller except the microcontroller used is 89c51 instead of 89c2051.
4.3.5 TTS256 IC
Features:
�Narrow 28-pin DIP
�Companion chip to Magnevation SpeakJet or the Savage Innovations SpeakGin
�Built in 600 Rule database convert English text to phoneme codes
�9600 N81 Baud Serial interface (TTL level)
�Generate speech from ASCII text Compatible with Basic Stamp, OOPic, Pic and any
processor with a serial port
25
The TTS256 is an 8-bit microprocessor programmed with letter-to-sound rules.
This built-in algorithm allows for the automatic real-time translation of English ASCII
characters into allophone addresses compatible with the Magnevation SpeakJet Speech
Synthesizer IC. Combine this with the SpeakJet to build a complete text-to- speech
solution.
Fig 4.8 TTS256 DIP
Table 4.2 TTS pin diagram
Pin # TTS256 Pin # SpeakJet
5 TX (sends serial data to host)
14 Vss (ground)
18 RX (received serial data from host)
20 Ready 15 Buffer Half Full
24 SJ_TX 10 RX
28 Vdd (+5v)
The TTS256 contains over 600 rules for pronouncing English text. It does a pretty good
job of the task with a less than 5% error rate in most sentences, it will mispronounce
some words. Often times, some creative spelling will help.
4.3.6 Speakjet IC
This little 18 Pin IC is exactly what we need to add speech and audio to our
project. All we need to add is +5V and a speaker.
26
Features:
�Programmable 5 channel synthesizer
�Natural phonetic speech synthesis
�DTMF and other sound effects
�Control of pitch, rate, bend and volume
�Programmable announcements
�Multiple modes of operation
�Simple interface to microcontrollers
�Simple, Stand Alone operation
�Three programmable digital outputs
�Internal 64 Byte buffer
�Internal programmable EEPROM
�Extremely low power consumption
�Low pin count
�Multiple case styles available
�Only need power and speaker to hear speech
Fig 4.9 Speakjet DIP
27
The SpeakJet is a completely self-contained, single chip voice and complex sound
synthesizer. It uses a mathematical sound algorithm to control an internal five channel
sound synthesizer to generate on-the-fly, unlimited vocabulary speech synthesis and
complex sound generation.
The SpeakJet is programmed with 72 speech elements (allophones), 43 sound effects, and
12 DTMF Touch Tones. Through the selection of these sounds and in combination with
the control of the pitch, rate, bend, and volume parameters, the user has the ability to
produce unlimited phrases and sound effects, with thousands of variations, any time. The
SpeakJet can be controlled simultaneously by logic changes on any one of its eight
Event Input lines, and by a single I/O line from a CPU (such as the OOPic or the Basic
Stamp) allowing for both CPU-Controlled and Stand- Alone operations.
Other features include an internal 64 byte input buffer, Internal Programmable EEPROM,
three programmable outputs, and direct user access to the internal five channel sound
synthesize
The interfacing figure of the TTS256 and Speakjet IC is as given below.
Fig 4.10 TTS256 and Speakjet Interfacing figure
28
4.3.7 Character LCD
Features:
�16 Characters x 2 Lines
�5 x 7 Dots with Cursor
�Built in Controller
�+5v Power Supply (Also Available for +3V)
�1/16 Duty Circle
Fig 4.11 Character LCD
Table 4.3 LCD Pin Diagram
29
4.3.8 HT6230 Infrared Remote Encoder
Features:
�Operating voltage: 2.4V~5.2V
�32 system codes, each system with
�64 command codes
�Programmable transmission codes
�Biphase transmission method
�Generated modulation output data(1/2 system frequency and 1/4 duty cycle)
�Single pin oscillator
�429kHz resonator system clock
�Test pins available
�28-pin SOP package
Applications:
�Televisions and video cassette recorder
�Microcontrollers
�Garage door controllers
30
Fig 4.12 HT6230 DIP
The HT6230 is designed as infrared remote encoder, usually applied to TV systems. A
total of 2048 different commands can be generated and arranged into 32 systems where
each system contains 64 different commands. There are 96 keys and to each key is
assigned one programmable code. The code is programmable by mask option. Legal and
illegal key operation can be distinguished.
31
5.1 Drawings of method using IR.
Fig 5.1 Right hand Microcontroller unit along with sensors
Fig 5.2 Left hand Microcontroller unit along with sensors
WORKING/IMPLEMENTATION Chapter-5
32
Fig 5.3 Neck Microcontroller unit along with
TTS256 and Speakjet
Fig 5.4 Finger open condition detailing
33
Fig 5.5 Finger closed condition detailing
Fig 5.6 Gesture operated T.V.
34
Fig 5.7 Front View of sensors
5.2 Working of the method using IR.
Referring now to the invention in more detail, In Fig 5.1, Fig 5.2, and Fig 5.4, each of the
rings 11,12,13,14 worn on each fingers and ring 10 worn on the thumb contains an IR
LED 17 that emits Infrared light and a photo-detector 18 which detects the Infrared light.
All these ring sensors are connected to Microcontroller and processing unit 16, 30
through connecting wires 15 which processes the data from all the sensors.
Referring now to Fig 5.4, the IR LED 17 emits the IR light 24. But since the finger is
open the IR light 24 does not reflect back to the photo-detector 18 and hence its
resistance is very high and voltage is low. Thus port of microcontroller which is
connected to this sensor will detect logic “0” in this case.
Referring now to Fig 5.5, the index finger is now closed thus the IR light 25 emitted by
the IR LED 17 is reflected back to the photo- detector from the tip of the finger. This
reflected ray 25 decreases the resistance of photodiode 18 and hence increases the
voltage. Thus, the port of microcontroller will detect logic “1” in this case. Thus, the
sensor ultimately works as a proximity sensor.
Therefore, in microcontroller and processing units 16, 30 if we keep on polling the ports
35
connected to all the sensors of all the fingers, we collect data periodically of fingers
which are open and fingers which are closed. The microcontroller and processing unit 30
will generate code corresponding to position of finger which is opened. This code is sent
to microcontroller 16 through serial communication via RF transmission. Also
microcontroller and processing unit 16 decodes this incoming code from microcontroller
and processing unit 30 and gets the position of each finger on left hand. Combining
this data with the data obtained via polling; the position of fingers on right hand
generates a specific code indicating a particular gesture (i.e. position of 10 fingers).
Every gesture will generate a unique code inside the microcontroller and processing unit
16 and it is compared with the predefined codes for valid gestures. Thus, gesture
recognition is done for valid gestures and further decision making and controlling can be
done by the microcontroller.
Referring now to Fig 5.7 for an example; if we want to gesture numerical number “2”,
then we can raise index finger and the middle finger. The ports connected to the
index finger ring 50 and middle finger ring 51 will have logic “1” and other 8 ports
connected to other 8 sensors 10,11,12,13,14,52,53,54 will have logic “0”. By
previously described procedure, microcontroller and processing unit 16 will be able to
generate a code for this gesture which will be compared with predefined codes
and matching of code will be done for the predefined code of numerical number
“2” and hence gesture of “2” can be interpreted.
Now Referring to Fig 5.3, it shows a case 19 hanging from the human neck 21, via a
string 20 across human shoulders 22.
Again Referring to Fig 5.3,if we set predefined gestures and corresponding codes for
every English alphabet from A-Z and 0-9,then whenever any of the above gesture is
made, the ANCII code of that particular alphabet is generated by microcontroller and
processing unit 16 and is sent to microcontroller and processing unit 60 via wireless RF
transmission. This microcontroller and processing unit is installed inside a case 19 which
can be worn on the neck. Microcontroller and processing unit 60 will send this ANCII
code to TTS 256 IC which converts ANCII codes of English text to allophones. This is
further interfaced to another IC to convert these allophones into electrical speech signal.
These signals are then given to loudspeaker 62 to create sound. Thus, Gesture such as
36
“C”;”A”;”T” can be converted into sound which spells “CAT”.
Also Referring to Fig 5.6, when the invention operated to control any other device
wirelessly, the microcontroller and processing unit 16 will generate codes related to a
particular gesture which can be sent via one of the sensor of the finger which is open via
IR transmission. The codes generated and sent via microcontroller and processing unit
16 will be same as generated and sent in any conventional remote controls. These codes
are then decoded at controlled device 27. Thus, different gesture can be assigned to
perform different operation to control any device ranging from Televisions and Air
conditioners to Robots and Wheel chairs.
5.3 Drawings of the method using TOUCH-PAD.
Fig 5.8 Front View of Left hand with touchpad
37
Fig 5.9 Front View of Right hand with touchpad
Fig 5.10 Position of touchpad for thumb.
38
5.4 Working/Implemantation using TOUCH-PAD
The above method is implemented with the motive of effective utilization of the
components. With the use of first method, the discharging of the battery was a major
issue, which can be solved by using sensor as a switch (i.e., as a touch contact).
Fig 5.8 shows front view of the left hand with its touchpad. As shown in the figure, the
sensors (touchpad) are placed on the palm and the tip of all the fingers are metal contact.
When these metal contacts make a contact with the touchpad on the palm, it works as a
close circuit and losing of contact makes it work as an open circuit and the further
processing of the gesture is carried out.
Fig 5.9 shows front view of the right hand with its touchpad. As shown in the figure, the
sensors (touchpad) are placed on the palm and the tip of all the fingers are the metal
contact. When these metal contacts make a contact with the touchpad on the palm, it
works as a close circuit and losing of contact makes it work as an open circuit and the
further processing of the gesture is carried out.
Fig 5.10 shows the separate connection made for thumb for which the touchpad is placed
on the left face of the index finger for better feasibility and flexibility. The thumb
connection is not shown in the Fig 5.8 and Fig 5.9 for the same reason and the same
connections are made in the actual hardware as shown in the figures.
39
5.5Algorithm of the C-program.
40
41
6.1 Proteus Simulation
Fig 6.1 Proteus simulation
RESULT ANALYSIS Chapter-6
42
6.2 Hardware results
Fig 6.2 Right hand module
43
Fig 6.3 Left hand module
44
Fig 6.4 Complete module
45
Fig 6.5 Breadboard implementation of TTS256 module
46
Thus we conclude our completed Final year project by developing an innovative idea
for gesture recognition .This project has given us the opportunity to work in many
technical domain ranging from optical electronics to wireless communication and c
programming. We learn the process of developing a raw innovative idea .This project
will surely be a boon to dumb people and will also help society to be more gesture
oriented .This project can be expanded further for many applications such as gesture
controlled wheelchair for disabled people, Gesture based Calculator and also can be
used for Home Automation. It
CONCLUSION Chapter-7
47
REFERENCES
1. Pranav Mistry , Paricia Maes , “Wearable Gesture Interface “ –USPTO Pub
No: US 2010/0199231 A1 (AUG 5 , 2010)
2. Walt Jung , “OP-AMP Application Handbook” , NEWNES Publication
3. B.P.Lathi , Zhi Ding, “ Modern Digital And Analog Communication System”,
2nd edi, Oxford University Press
4. Gerd Keiser, “ Optical Fiber Communication” , 4th edi, Mcgraw-hill
Publication
5. Muhammad Ali Mazidi, Janice Gillispie Mazidi, Rolind McKinlay, “ 8051
Microcontroller and Embedded system”, 2nd edi, Pearson Pub
6. Balaguruswamy, “ANSI C”,2nd edi, The Mcgraw-hill Pub
7. www.sparkfun.com,“Speakjet Natural speech and complexsound synthesizer User’
Manual”, Revision 1.0
8. www.sparkfun.com, “ TTS256 Text to speech processor datasheet”,
9. www.sunrom.com, “ST3654 Serial interface IC for RF transceiver based on
CC2510 Datasheet” can be made more reliable, portable and accurate.
48
REWARDS
1. We have been awarded a provisional patent for the same innovation mentioned
throughout this report. “Under "PATENT PENDING STATUS" at USPTO.
App No - 61514913 Conformation No: 6538”
2. Our project has also been selected in a National Level technical Project/Innovation
Event JED-i Project challenge conducted by IISC BANGLORE and we have been
shortlisted in top 20 projects for the same innovation by JED-i team.