8/2/2019 Documentation_Vision Based Game Development Using Human
1/70
Abstract
A Human Computer Interface (HCI) System for playing games is designed here for more
natural communication with the machines. The system presented here is a vision-based systemfor detection of long voluntary eye blinks and interpretation of blink patterns for
communication between man and machine. This system replaces the mouse with the human
face as a new way to interact with the computer. Facial features (nose tip and eyes) are detected
and tracked in real-time to use their actions as mouse events. The coordinates and movement of
the nose tip in the live video feed are translated to become the coordinates and movement of the
mouse pointer on the application. The left/right eye blinks fire left/right mouse click events.
The system works with inexpensive USB cameras and runs at a frame rate of 30 frames per
second.
1
8/2/2019 Documentation_Vision Based Game Development Using Human
2/70
Chapter 1
INTRODUCTION
1.1 Purpose of the project
In the past few years high technology has become more progressed, and less expensive. With
the availability of high speed processors and inexpensive webcams, more and more people
have become interested in real-time applications that involve image processing. One of the
promising fields in artificial intelligence is human computer interaction which aims to use
human features (e.g. face, hands) to interact with the computer. One way to achieve that is to
capture the desired feature with a webcam and monitor its action in order to translate it to some
events that communicate with the computer.
The current evaluation of computer technologies has enhanced various applications in
human-computer interface. Face and gesture recognition is a part of this field, which can be
applied in various application such as in robotic, security system, drivers monitor, image
processing.
Since human face is a dynamic object and has a high degree of variability. Face detection
can be classified as two categories feature-based approach and image-based approach. The
techniques in the first category makes used of apparent properties of face such as geometry,
skin color, and motion. Even feature-based technique can achieve high speed in face detection,
but it also has problem in poor reliability under lighting condition. For second category, the
image-based approach takes advantage of current advance in pattern recognition theory. Most
of the image-based approach applies a window scanning technique for detecting face, whichrequires large computation. Therefore, by using only image-based approach is not suitable
enough in real-time application.
In order to achieve high speed and reliable face detection system, we propose the
method combine both feature-based and image-based approach to detect the point between the
eyes by using SSR[1].The proposed SSR filter, which is the rectangle divided into 6 segments,
operates by using the concept of bright-dark relation around BTE [2] area. We select BTE [2]
as face representative because it is common to most people and easy to find for wide range of
2
8/2/2019 Documentation_Vision Based Game Development Using Human
3/70
face orientation. BTE [2] has dark part (eyes and eyebrows) on both sides, and has comparably
bright part on upper side (forehead), and lower side (nose and cheekbone).
We use an intermediate representation of image called integral image to calculate sums
of pixel values in each segment of SSR [1] filter. Firstly, SSR [1] filter is scanned on the image
and the average gray level of each segment is calculated from integral image. Then, the bright-
dark relations between each segment are tested to see whether its center can be a candidate
point for BTE [2]. Next, the camera is used to find the distance information and the suitable
BTE [2] template size. Then, the BTE [2] candidates are evaluated by using a template of BTE
[2]. Finally the true BTE [2], nose tip can be detected.
Using the co-ordinate of mouse tip as pointer and eye blink fires the left/right mouse clickevents.
1.2 Objective
Our system is capable of replacing the traditional mouse with the human
face a new way to interact with the computer.
Facial features (nose tip and eyes) are used to interact with computer.
The nose tip acts as a mouse pointer and the left/right eye blinks fire
left/right mouse click events.
3
8/2/2019 Documentation_Vision Based Game Development Using Human
4/70
Chapter 2
LITERATURE REVIEW
2.1 Existing System
HCI (Human Computer Interface) aims to use human features (e.g. face, hands) to interact with
the computer. One way to achieve that is to capture the desired feature with a webcam and
monitor its action in order to translate it to some events that communicate with the computer.
Traditional Mouse
Wireless Mouse
Roller bar Mouse
2.1.1 Traditional Mouse
In computing, a mouse (plural mice or mouses) functions as a pointing device by detecting
two-dimensional motion relative to its supporting surface. Physically, a mouse consists of a
small case, held under one of the user's hands, with one or more buttons. It sometimes features
other elements, such a "wheels", which allow the user to perform various system-dependent
operations, or extra buttons or features can add more control or dimensional input. The mouse'smotion typically translates into the motion of a pointer on a display, which allows for fine
control of a Graphical User Interface.The name mouse originated at the Stanford Research Institute derives from the
resemblance of early models (which had a cord attached to the rear part of the device,
suggesting the idea of a tail) to the common mouse.
Fig 2.1Wired mouse
2.1.2 Wireless Mouse
4
8/2/2019 Documentation_Vision Based Game Development Using Human
5/70
A Wireless Mouse is a computer mouse that communicates with the CPU via RF wireless
ID technology instead of via a plug-in cord.
Fig 2.2 Wireless mouse
Technical features of the mouse are
Touch-sensitive top shell
360 degree enabled clickable scroll ball
Force-sensing side "squeeze" buttons
Optical tracking in wired version or laser tracking in wireless version
Compatible with Macintosh and PCs
Programmable functions for the four buttons
Easter egg: when held parallel above a surface, the shape of the light
projected from the wired USB mouse resembles the head of a mouse
2.1.3 Roller Mouse
Roller bars are one of the more esoteric computers pointing devices, consisting essentially of a
round, rubber coated bar situated between the computer keyboard and the user, who
manipulates the screen cursor through a combination of rolling the bar on its axis and sliding it
sideways in its housing.
The operation is a lot more intuitive than it sounds, and the theory behind it is that
roller bars reduce physical stress, especially on the elbows and shoulders, by eliminating the
reaching necessary when using any movable mouse - conventional or ergonomic.
5
8/2/2019 Documentation_Vision Based Game Development Using Human
6/70
Contour Design's Roller mouse roller bar station is designed to be used with standard
computer keyboards that have a rectangular form factor and a straight edge on the user side.
Fig 2.3 Roller bar mouse
The Roller Mouses roller bar is contained in a full-width housing that incorporates a
console of click buttons and a scroll wheel flanked by padded wrist rests. An apron extends
from the control housing for the keyboard to rest on.
The Roller Mouse controls the cursor without the need to grip or reach for a mouse, and
requires minimal space. Button controls are centrally located beneath the space bar. The Roller
Mouse is also a solution for workspaces where there is no room for a conventional mouse.
The roller bar can be manipulated using either hand or whatever combination of
thumb, fingers, or palm control is most comfortable. This facility for spreading mousing stress
between both hands and various digits is a key element in reducing the overall stress on the
user's body.
Clicking may also be executed in several ways, either with the three buttons andclickable scroll wheel or by pressing down on the roller bar itself, which is clickable as well.
Button assignments re programmable, with defaults as follows.
6
8/2/2019 Documentation_Vision Based Game Development Using Human
7/70
Fig 2.4 Working of roller bar
Disadvantage of Roller Mouse
Clickable roller bar: primary or left button
Left click button: primary or left button Middle click button: preprogrammed for double click
Right click button: secondary or right button (i.e.: activates contextual
menus)
Scroll wheel button: auxiliary or middle button
We interchange hand from keyboard to mouse while working in
computer
It takes large time to work
Hand disabilities persons are unable to work properly
2.2 Proposed System
Our system is capable of replacing the traditional mouse with the human face - a new way to
interact with the computer. Facial features (nose tip and eyes) are used to interact with
computer. The nose tip acts as a mouse pointer and the left/right eye blinks fire left/right mouse
click events .Only external device that the user needs is a webcam that feeds the program with
the video stream .The people having hand disabilities are able to work.
Chapter 3
7
8/2/2019 Documentation_Vision Based Game Development Using Human
8/70
SYSTEM REQUIREMENTS
3.1 Introduction
The Java Media Framework (JMF) is a recent API for Java dealing with real-time multimedia
presentation and effects processing. JMF handles time-based media, media which changes
with respect to time. Examples of this are video from a television source, audio from a raw-
audio format file and animations. The beta JMF 2.0 specification will be used for this report,
as they currently reflect the features that will appear in the final version.
3.2 Stages
The JMF architecture is organized into three stages:
Fig 3.1 JMF Architecture
During the input stage, data is read from a source and passed in buffers to the
processing stage. The input stage may consist of reading data from a local capture device (such
as a webcam or TV capture card), a file on disk or stream from the network.
The processing stage consists of a number of codecs and effects designed to modify the
data stream to one suitable for output. These codecs may perform functions such as
compressing or decompressing the audio to a different format, adding a watermark of some
kind, cleaning up noise or applying an effect to the stream (such as echo to the audio).
8
8/2/2019 Documentation_Vision Based Game Development Using Human
9/70
Once the processing stage has applied its transformations to the stream, it passes the
information to the output stage. The output stage may take the stream and pass it to a file on
disk, output it to the local video display or transmit it over the network.
3.3 Component Architecture
JMF is built around component architecture. The components are organized into a number of
main categories:
Media handlers
Data sources
Codecs/Effects
Renderers
Mux/Demuxes
3.3.1 Media Handlers
Media Handlers are registered for each type of file that JMF must be able to handle. To
support new file formats, a new Media Handler can be created.
3.3.2 Data Source
A Data Source handler manages source streams from various inputs. These can be for network
protocols, such as http or ftp, or for simple input from disk.
3.3.3 Codecs/Effects
Codecs and Effects are components that take an input stream, apply a transformation to it and
output it. Codecs may have different input and output formats, while Effects are simple
transformations of a single input format to an output stream of the same format.
3.3.4 Renderers
A renderer is similar to a Codec, but the final output is somewhere other than another stream.
A VideoRenderer outputs the final data to the screen, but another kind of renderer could output
to different hardware, such as a TV out card.
9
8/2/2019 Documentation_Vision Based Game Development Using Human
10/70
3.3.5 Muxs / Demuxes
Multiplexers and Demultiplexers are used to combine multiple streams into a single stream or
vice-versa, respectively. They are useful for creating and reading a package of audio and
video for saving to disk as a single file, or transmitting over a network.
3.4 System Specification
The Hardware and software required for implementing this project are listed below.
3.4.1 Hardware Requirement (Minimum Requirement)
Processor Pentium III 700MHz
CPU 500 MHz
RAM 128 MB
Monitor 17 Samsung color monitor
Hard Disk 40 GB
Keyboard Standard Keyboard with 104 Keys
Mouse Serial mouse
Camera Web Camera
3.4.2 Software Requirement
Tools Java 5.0, Java Media Framework 2.1.1e
Platform Windows 98/2000, Windows XP
10
8/2/2019 Documentation_Vision Based Game Development Using Human
11/70
Chapter 4
DESIGN
4.1 System Designing
The skeleton of the entire process is prepared in this phase
The scheduling and interactivity of the system is completed in designing
The work structure including look and feel is generated for the system
4.1.1 Input Design
Input design is the process of converting user-originated inputs to a computer based format.
Input design of the Eye encompasses various controls such as buttons and menu items. The
refresh and stop buttons are used to refresh and stop receiving the video frames from the video
input device .The enable interface button is used to deactivate the mouse and act as a interface
for interacting with the system. The face detection button is used to detect the face candidates
(nose tip and eyes).The menu items such as file, help are used to perform normal operations
such as exit and to know the information about the project.
4.1.2 Output Design
Output design generally refers to the results generated by the system and it is a direct source of
information to the user. For many end users, output is the main reason for developing the
system and the basis on which they evaluate the usefulness of the application. The objective of
a system finds its shape in terms of the output.
The output of this project tracks and identifies the human facial features (nose tip and
eyes).
4.2 Modules
Module 1
Live Video Capture
Face Detection Find Face Candidates
Module 2
11
8/2/2019 Documentation_Vision Based Game Development Using Human
12/70
Extract Between The Eyes
Find Nose Tip
Module 3
Face tracking
Module 4
Front end
4.3 Block Diagram
Fig 4.1 Block diagram for eye controlled interaction
In video frame module we capture the live video, track the nose tip by using SSR filterand the ROI [4] concepts. Then BTE [2] is tracked, the sector is binarized with a certain
threshold. The cluster of the binarized sectors is found. Detecting eye brows is done by using
Squared Difference (SSD).Detect eye blink by using Motion Detection algorithm. For tracking
the eyes we use SSR [1] filter. Finally refining the nose tip is done by using nose bridge
detection with SSR filter and horizontal profile.
12
8/2/2019 Documentation_Vision Based Game Development Using Human
13/70
4.4 Flow Diagram
Fig 4.2 Flow Diagram for eye controlled interaction
SSR FILTER
STERO CAMERA TO FIND THE
DISTANCE INFORMATION
AVERAGE BETWEEN THE EYE
TEMPLATE MATCHING
EYE DETECTION
BETWEEN THE EYE POINT
DETECTION
CANDIDATE POINT
BETWEEN THE EYE
TEMPLATE
13
8/2/2019 Documentation_Vision Based Game Development Using Human
14/70
4.5 Data Flow Diagram
Fig 4.3 Data Flow Diagram 1
14
8/2/2019 Documentation_Vision Based Game Development Using Human
15/70
Fig 4.4 Data Flow Diagram 2
Fig 4.5 Data Flow Diagram 3
15
8/2/2019 Documentation_Vision Based Game Development Using Human
16/70
Chapter 5
IMPLEMENTATION
5.1 Problem Statement
In traditional mouse or in any wireless mouse, movement of the hand is necessary to select or
deselect the object. This becomes difficult for those with hand disabilities.
5.2 Project Developed
In our project we overcome this by using facial features to interact with computer i.e., nose tip
is tracked and it is used as mouse pointer and left and right blinks fire left and right
respectively. Our project consist 3 modules.
5.3 Module1
5.3.1 Live Video Capture
The first module is to capture the Live Image using the Web Cam and to detect a face in the
captured Image Segment.
5.3.2 Face Detection Module
Face Detection can be categorized as
Feature Based Method
Image Based Method
5.3.2.1 Feature Based Method
This techniques makes use of
apparent properties of face such as geometry, skin color, and motion
Lack of Robustness against head
rotation ,scaling and problem under lightning condition
5.3.2.2 Image Based Method
The image of
interest with a window that looks for faces at all scales and locations.
16
8/2/2019 Documentation_Vision Based Game Development Using Human
17/70
Face detection
implies pattern recognition, and achieves it with simple methods such as template
matching
Not suited for
real time application
In order to achieve the high speed and reliable face detection system, we
combine these two techniques by using Six-Segmented Rectangular filter (SSR Filter) in figure
5.1
Fig 5.1 SSR Filter
The sum of pixels in each sector is denoted as S along with the sector number. The use
of this filter will be explained in detail in the face detection algorithm.
5.3.2.3 Integral Image
In order to facilitate the use of SSR filters an intermediate image representation called integral
image will be used. In this representation the integral image at location x, y contains the sum of
pixels which are above and to the left of the pixel x, y (see fig. 5.2)
17
8/2/2019 Documentation_Vision Based Game Development Using Human
18/70
Fig 5.2 Integral Imageii: Integral image, i: Pixel value.
With this representation calculating the sectors of the SSR filter becomes fast and easy. No
matter how big the sector is, we will need only 3 arithmetic operations to calculate the sum of
pixels which belong to it (see fig.5.2).Sector = D B C + A (5.1)
So each SSR filter requires 6*3 operations to calculate it.
5.3.3 Find Face Candidates Module
The human face is governed by proportions that define the different sizes and distances
between facial features. We will be using these proportions in our heuristics to improve facial
features detection and tracking.
5.3.3.1 Face detection general steps
We will be using feature based face detection methods to reduce the area in which we are
looking for the face, so we can decrease the execution time. To find face candidates the SSR
[1] filter will be used in the following way:
At first we calculate the integral image by making a one pass over the video frame
using these equations:
s(x, y) = s(x, y-1) + i(x, y) (5.2)
ii(x, y) = ii(x-1, y) A+ s(x, y) (5.3)
Where s(x, y) is the cumulative row sum, s(x,-1) = 0, and ii(-1, y) =0. Figure shows an ideal
location of the SSR [1] filter, where its center is considered as a face candidate.
18
8/2/2019 Documentation_Vision Based Game Development Using Human
19/70
Fig5.3 Finding face candidate
(x, y) is the location of the filter (upper left corner). The plus sign is the center of the filter
which is the face candidate.
We can notice that in this ideal position the eyes fall in sectors S1 and S3, while the
nose falls in sector S5. Since the eyes and eye brows are darker than the BTE and the cheek
bones, we deduce that :
S1 < S2 && S2 > S3 (5.4)
S1 < S4 && S3 < S6 (5.5)
So in order to find face candidates we place the upper left corner (see fig.5.3) of the
SSR filter on each pixel of the image (only on pixels where the filter falls entirely inside the
bounds of the image) .For each location (x, y) we check equations (5.4,5.5); if the conditions
are fulfilled then the center of the filter will be considered as a face candidate if its a and b
values fulfill equations 5.4 and 5.5 (skin pixel thresholds).Eventually the candidates will group
in clusters.
5.4 Module 2
5.4.1 Extract BTE (Between the Eyes) Module
In order to extract BTE templates we need to locate pupils candidates. Left and right pupils
candidates will fall in sectors S1, S3. For each of the earlier mentioned sectors the following
steps are applied to find the pupils. To find the pixels that belong to a dark area (in our case the
19
8/2/2019 Documentation_Vision Based Game Development Using Human
20/70
pupil); the sector is binarized with a certain threshold. The clusters of the binarized sector are
found.
1. If the thresholding produces only one cluster like sector S3 (right eye) in figure 5.3, it is a
large probability that the clusters of the eye brow and the pupil has unified in one, so in this
case it is likely to look for the pupil in the lower half of the sector cause as we already
mentioned the upper half is probably the eye brow. So what we are going to do is to calculate
the area of the part of the cluster which is in the lower half of the sector, if it is larger than a
specified threshold then the center of the lower part is considered as the pupil; if not the same
will be applied to the upper half because it is possible that the cluster that was found is the
pupil cluster alone (without the eye brow cluster) and it fell in the upper half. In case of failing
to find a pupil in the upper half this sector will be Omitted and no pupil is found.
2. If there are multiple clusters like sector S1 (left eye), we need to find the cluster that is:
The largest: some clusters are the result of video noise and they are small.
The darkest: the pupil is darker from other features in the sector
The closest: to the darkest pixel of the sector: the eye pupil is black and we need to find
the cluster that contains the pupil or is the closest to it.
These criteria are applied in the lower half of the sector first, because as already
mentioned we need to avoid picking the eye brow cluster as the pupil, if the chosen cluster is
larger than a certain threshold its center will be considered as the pupil, if not, we will do the
same in the upper half, in case of failing to find a pupil in the upper half this sector will be
omitted and no pupil is found. If we didnt find a left or right pupil candidate the cluster will be
skipped and no longer considered as a face candidate.
5.4.2 Find Nose Tip Module
Now that we located the eyes, the final step is to find the nose tip. From the following figure
we can see that the blue line defines a perfect square of the pupils and outside corners of the
mouth; the nose tip should fall inside this square, so this square becomes our region of interest
in finding the nose tip.
20
8/2/2019 Documentation_Vision Based Game Development Using Human
21/70
Fig 5.4 Location of ROI
So the first step is to extract the ROI [4], in case the face was rotated we need to rotate
the ROI [4] back to a horizontal alignment of the eyes. The nose tip has a convex shape so it
collects more light than other features in the ROI because it is closer to the light source. Using
the previous idea we tried to locate the nose tip with intensity profiles in horizontal intensity
profiles we add vertically to each line the values of the lines that precedes it in the ROI [4], so
since that the nose bridge is brighter than the surrounding features the values should
accumulate faster at the bridge location; in other words the maximum value of the horizontal
profile gives us the x coordinate of the nose tip. In vertical intensity profiles we add
horizontally to each column the values of the columns that precedes it in the ROI [4] (see fig.
5.5); the same as in the horizontal profile, the values accumulate faster at the nose tipposition
so the maximum value gives us the y coordinate of the nose tip. From both, the horizontal and
vertical profiles we were able to locate the nose tip position, but unfortunately this method did
not give accurate results because there might be several max values in a profile that are close to
each other, and choosing the correct max value that really points out the coordinates of the nose
tip is a difficult task.
So instead of using the intensity profiles alone, we will be applying the following
method.
At first we need to locate the nose bridge and then we will find the nose tip on that
bridge. As earlier mentioned the nose bridge is brighter than surrounding features, so we will
use this criterion to locate the nose-bridge-point (NBP) on each line of the ROI.
21
8/2/2019 Documentation_Vision Based Game Development Using Human
22/70
We will be using an SSR filter to locate the NBP [5] candidates in each ROI [4] line.
The width of the filter is set to the half of the distance between the eyes, because from figure
5.5 we can notice that the line (nose width) is equal to the half of the line (distance between the
eyes).
5.4.2.1 SSR filter to locate nose bridge candidates
After calculating the integral image of the ROI [4], each line of it will be scanned with this
filter; we remember that the nose bridge is brighter than the regions to the left and right of it; in
other words the center of the SSR filter is considered as an NBP [5] candidate if the center
sector is brighter than the side sectors:
S2 > S1 (5.5)
S2 > S3 (5.6)
In each line we might get several NBP candidates, so the final NBP will be the candidate
that has the brightest S2 sector. In order to avoid picking some bright video noise as the NBP
[5] we will be using the horizontal intensity profile; so instead of applying the SSR [1] filter to
a line of the ROI we will be applying it to the horizontal profile calculated from the first line to
the line that we are dealing with, because as already mentioned the values will accumulate
faster at the nose bridge location, so by using the horizontal profile we are sure that we are
picking the right NBP
Candidate not some bright point caused by noise; of course the results will get more
accurate as we reach the last line of the ROI because the accumulation at the nose bridge
location will get more obvious.
Fig 5.6 Nose bridge detection
22
8/2/2019 Documentation_Vision Based Game Development Using Human
23/70
ROI is enlarged only for a clearer vision. Now that we located the nose bridge we need
to find the nose tip on that bridge. Since each NBP represents the brightest S2 sector on the line
it belongs to, and that S2 sector contains the accumulated vertical sum of the intensities in that
sector from the first line to the line it belongs to, we will be using this information to locate the
nose tip. Nose trills are dark areas, and the portion that they add to the accumulated sum in the
horizontal profile is smaller than the contribution of other areas; in other words each NBP will
add with its S2 sector a certain amount to the accumulated sum in the horizontal profile, but the
NBP at the nose trills location will add a smaller amount (S2 sector of the NBP at the nose trills
location is darker than S2 sector of other NBPs), so if we calculate the first derivate of the
values of NBPs S2 sectors (the first derivate of the values of the maximum value of the
horizontal profile at each ROI line) we will notice a local minima at the nose trills location (see
fig. 5.6); by locating this local minima we take the NBP that corresponds to it as the nose trillslocation, and the next step is to look for the nose tip above the nose trills.
Since the nose tip is brighter than other features it will donate with its S2 sector to the
accumulated sum more than other NBPs [5], which means a local maxima in the first derivate
(see fig. 5.6); so the location of the nose tip is the location of the NBP that corresponds to the
local maxima that is above the local minima in the first derivate.
5.5 Module 3
5.5.1 Face Tracking Module
The nose tip is tracked to use its movement and coordinates as the movement and coordinates
of the mouse pointer. The eyes are tracked to detect their blinks, where the blink becomes the
mouse click.
The tracking process is based on predicting the place of the feature in the current
frame based on its location in previous ones; template matching and some heuristics are applied
to locate the features new coordinates.
5.6 Module 4
5.6.1 Front End
23
8/2/2019 Documentation_Vision Based Game Development Using Human
24/70
This light weight eye controlled computer has a front end which is created using SWINGS and
AWT. Java swing has many built in APIs for creating the front end. The front end of the eye
controlled computer provides for a good user interface.
Swing is a GUI toolkit for Java. It is one part of the Java Foundation Classes (JFC).Swing includes graphical user interface (GUI) widgets such as text boxes, buttons, split-panes,
and tables. Swing widgets provide more sophisticated GUI components than the earlier
Abstract Window Toolkit. Since they are written in pure Java, they run the same on all
platforms, unlike the AWT which is tied to the underlying platform's windowing system. Swing
supports pluggable look and feel not by using the native platform's facilities, but by roughly
emulating them. This means you can get any supported look and feel on any platform. The
disadvantage of lightweight components is slower execution. The advantage is uniform
behavior on all platforms.
Advantages of Swings:-
Swing is a platform independent, Model-View-ControllerGUI framework for Java. It follows a
single-threaded programming model, and possesses the following traits:
Platform independence: Swing is platform independent both in terms of its expression
(Java) and its implementation (non-native universal rendering of widgets).
Extensibility: Swing is a highly partitioned architecture which allows for the 'plugging'
of various custom implementations of specified framework interfaces: Users can
provide their own custom implementation(s) of these components to override the
default implementations. In general, Swing users can extend the framework by:
extending existing (framework) classes; providing alternative implementations of core
components.
Component-Oriented: Swing is a component-based framework. The distinction
between objects and components is a fairly subtle point: concisely, a component is a
well-behaved object with a known/specified characteristic pattern of behavior. Swing
objects asynchronously fire events, have 'bound' properties, and respond to a well
known set of commands (specific to the component.) Specifically, Swing components
are Java Beans components, compliant with the Java Beans Component Architecture
specifications.
24
8/2/2019 Documentation_Vision Based Game Development Using Human
25/70
Customizable: Given the programmatic rendering model of the Swing framework, fine
control over the details of rendering of a component is possible in Swing. As a general
pattern, the visual representation of a Swing component is a composition of a standard
set of elements, such as a 'border', 'inset', decorations, etc. Typically, users will
programmatically customize a standard Swing component (such as a JTable) by
assigning specific Borders, Colors, Backgrounds, etc., as the properties of the
component. The core component will then use these properties (settings) to determine
the appropriate renders to use in painting its various aspects. However, it is also
completely possible to create unique GUI controls with highly customized visual
representation. (Swing components support transparent rendering.)
Configurable: Swing's heavy reliance on runtime mechanisms and indirect
composition patterns allow it to respond at runtime to fundamental changes in its
settings. For example, a Swing-based application can change its look and feel at
runtime (from say MacOS look and feel to a Windows XP look and feel). Further, users
can provide their own look and feel implementation, which allows for uniform changes
in the look and feel of existing Swing applications without any programmatic change to
the application code.
Lightweight UI: The magic of Swing's configurability is also due to the fact that it does
NOT use the native host OS's GUI controls for representation, but rather 'paints' its
controls programmatically, through the use of Java 2D api's. Thus, a Swing component
does NOT have a corresponding native OS GUI 'peer', and is free to render itself in any
way that is possible with the graphics APIs.
AWT, Abstract Windowing Toolkit, is a Javas original way of working with graphics.
AWT components are now called heavyweight components because of their significant use of
system resources. AWT is now supplemented with the Swing Package. AWT provides the
foundation of graphics work in Java, and even the Swing Package is based on AWT.
Text Fields
Text fields are the basic text handling components of the AWT. These components handle a
one dimensional string of text; they let you display text, let the user enter text, allow you to
25
8/2/2019 Documentation_Vision Based Game Development Using Human
26/70
take passwords by masking typed text, allow you to read the text the user has entered, and most
fundamental AWT components.
Buttons
Buttons provide users with a quick way to start some action-all they have to do is click them.
Every user is familiar with buttons, and we have already taken a look behind the scenes on how
buttons work in code when discussing event handling. You can give buttons a caption, such as
Click Me! when the user does click the button. Your code if you have registered to handle
events from the button.
Checkboxes
Checkboxes are much like, buttons, except that they are dual state, which means they canappear as selected or unselected. when selected ,they display a visual indication of some kind
,such as a checkmark or an x(it varies by operating system in AWT programming, which is
one of the reason sun introduced swing, which can display components with the same look
across many operating systems), the user can check a check box to select an option of some
kind (such as the items on sandwich),to enable automatic spell checking, or to enable
automatic spell checking, or to enable printing while he is doing something else. You use
checkbox to let the user select nonexclusive options; for example, both automatic spell
checking and background printing may be enabled at the same time.
Radio buttons
You let the user select one of a set of mutually exclusive options using options using radio
buttons. Only one of a set of option buttons can let the user select a printing color or the day of
the week. In AWT radio buttons are actually a type of checkbox, and when selected, they
display a round dot or a clicked square or some other indication (again, the visual indication
depends on the operating system).you use radio buttons in groups.
Layouts
We have just added components to applets and applications using the add method is actually a
method of the default layout manager-the flow layout manager is res possible for arranges
components in AWT applets, by default. The flow layout manager arranges much like a word
processor might arrange words-across the page and then wrapped to the next line as needed
,creating what sun calls a flow of components. you will see that you can customize flow
layouts to some extend ,such as left center, or right-aligning components .however ,the
26
8/2/2019 Documentation_Vision Based Game Development Using Human
27/70
limitation of flow layout are clear, especially if you are counting on components maintaining
some position with respect to others, because if user resizes your applet or application ,the
components will all move around. On the other hand, there are other AWT layout manager
(and quite a few new ones in swing), and we all cover the AWT grid, border, card, and grid bag
layout managers.
Why cant you just set where a components goes and forget it? Novice java
programmers are often frustrated by having to deal with AWT layout mangers and want to
know when they cant just give the coordinates for the components they wants to use and be
done with it. In fact, you can, although youre responsible for handling the case where
windows are resized and components should be moved to position components where you want
them, you can indicate that you want no layout manager at all. Then you can size and locatecomponents as you want them, using adds to display them, like this:
Setlayout(null);
text1 = new textfield(20);
text1.setsize(200,50);
text1.setlocaton(20,20);
add(text1);
This adds a text field of size (200, 50) at location (20, 20) in a container such as an
applet window. Therefore, as you can add components to containers without any layout
manager at all, something thats useful to bear in mind if the AWT layouts frustrate you too
much.
One very useful AWT container to use with layouts is the panel components you can
arrange components in a panel and then add the panel, itself, to the layout of an applet or
application.
AWT vs. SWING
Theres a fundamental difference philosophical difference between AWT and swing
components. Each AWT components gets its own operating platform window (and therefore
ends up looking like a standard controlling that operating platform). In extended programs, a
large number of such windows shows performance and uses up a great deal of memory .such
components have come to be heavyweight. Swings controls, on the other hand, are simply
27
8/2/2019 Documentation_Vision Based Game Development Using Human
28/70
drown as images in their containers and dont have an operating platform window of their own
at all, so they use far fewer system resources .therefore, theyre called lightweight components.
All swings components are derived from the Jcomponents class, and this class is in
turn, derived from the AWT container class, which has no heavyweight itself, so Jcomponent is
a lightweight class. Jcomponent adds a tremendous amount of programming support to the
AWT components class. Note that because all swings components are derived from
Jcomponent, and Jcomponent are derived from the AWT container class, all swing components
are also AWT components in that way, and you can mix AWT controls with swing controls in
your programs.
Not all swing components are lightweight; to display anything windowed environment,
you need some operating system windows, if only to draw lightweight controls in. For thatreason, swing supports these heavyweight classes; JFrame, JDialog, Jwindow, and JApplet.
Applet class and AWT application using the Frame class, you build swing applets on
the JApplet class and swing applications using the JFrame.
28
8/2/2019 Documentation_Vision Based Game Development Using Human
29/70
Chapter 6
SOURCE CODE
Application.Java
import javax.swing.UIManager;
import java.awt.*;
import c.WaitFrame;
public class Application
{
boolean packFrame = false;
//Construct the application
public Application()
{
WaitFrame wFrame = new WaitFrame();
Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize();
wFrame.setLocation((screenSize.width - wFrame.getWidth()) / 8, (screenSize.height -
wFrame.getHeight()) / 8);
wFrame.setUndecorated(true);
wFrame.getContentPane().setCursor(Cursor.getPredefinedCursor(Cursor.WAIT_CURSOR));
29
8/2/2019 Documentation_Vision Based Game Development Using Human
30/70
wFrame.show();
Frame frame = new Frame(wFrame);
//Validate frames that have preset sizes
//Pack frames that have useful preferred size info, e.g. from their layout
if (packFrame)
{
frame.pack();
}
else
{
frame.validate();}
//Center the window
Dimension frameSize = frame.getSize();
if (frameSize.height < screenSize.height)
{
frameSize.height = screenSize.height;
}
if (frameSize.width < screenSize.width)
{
frameSize.width = screenSize.width;
}
frame.setLocation((screenSize.width - frameSize.width)/2, (screenSize.height -
frameSize.height)/2 );
frame.setVisible(true);
}
//Main method
public static void main(String[] args)
{
try
{
UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
}
30
8/2/2019 Documentation_Vision Based Game Development Using Human
31/70
catch(Exception e) {
e.printStackTrace();
}
new Application();
}
}
ConnectedComponents.java
import java.util.*;
import java.awt.*;
public class ConnectedComponents
{
private int labels[],cluster;
private int label;
private int clustersMembers[][];
public int counters[],indecies[],clusters[];
public ConnectedComponents(int labels[],int clustersMembers[][])
{
label = 0;
this.labels = labels;
this.clustersMembers = clustersMembers;
}
public int findClustersLabels(int x,int y,int width)
{
//The algorithm is the following :
//If the pixel has no neighbors( i.e in a binary image: all neighbors are 0s ) assign a new label
to it
//If the pixel has one neighbor take it's label
31
8/2/2019 Documentation_Vision Based Game Development Using Human
32/70
//If the pixel has more than one neighbor take one of the labels and mark the labels as similar
int w,nw,n,ne;
if(y == 0)
{
if(x == 0)
{
label++;
clustersMembers[x][y] = label;
labels[label] = Integer.MIN_VALUE;
}
else
{if(clustersMembers[x-1][y] == 0)
{
label++;
clustersMembers[x][y] = label;
labels[label] = Integer.MIN_VALUE;
}
else
clustersMembers[x][y] = clustersMembers[x-1][y];
}
}
else
{
n = clustersMembers[x][y-1] ;
if( x+1 == width )
ne = 0;
else
ne = clustersMembers[x+1][y-1];
if(x != 0)
{
nw = clustersMembers[x-1][y-1];
w = clustersMembers[x-1][y];
if( (w == 0) && (nw == 0) && (n==0) && (ne==0) )
{
32
8/2/2019 Documentation_Vision Based Game Development Using Human
33/70
label++;
clustersMembers[x][y] = label;
labels[label] = Integer.MIN_VALUE;
}
else
{
if (w != 0)
{
clustersMembers[x][y] = w;
if(nw != 0)
{
if( nw != w )labels[nw] = w;
if( (n==0) && (ne!=0) && (nw != ne) && (labels[nw] != ne) )
labels[ne] = nw;
}
else
{
if(n != 0)
{
if( w != n)
labels[n] = w;
}
else
if( (ne != 0) && (ne != w) && (labels[w] != ne) )
labels[ne] = w;
}
}
else
{
if (nw != 0)
{
clustersMembers[x][y] = nw;
if( (n==0) && (ne!=0) && (ne !=nw) && (labels[nw] != ne) )
labels[ne] = nw;
33
8/2/2019 Documentation_Vision Based Game Development Using Human
34/70
}
else
if (n != 0)
clustersMembers[x][y] = n;
else
if (ne != 0)
clustersMembers[x][y] = ne;
}
}
}
else
{if( (n==0) && (ne==0) )
{
label++;
clustersMembers[x][y] = label;
labels[label] = Integer.MIN_VALUE;
}
else
{
if( n != 0)
clustersMembers[x][y] = n;
else
clustersMembers[x][y] = ne;
}
}
}
labels[label+1] = Integer.MAX_VALUE; //a flag that shows where the array ends
return label;
}
////////////////////////////////////////
public void findRootLabels() //For equivalent labels, find the representitive label
{
34
8/2/2019 Documentation_Vision Based Game Development Using Human
35/70
clusters = new int[label + 2];
cluster = 0;
if( label == 1 ) //one label found
{
cluster++;
clusters[cluster] = 1;
clusters[cluster + 1] = Integer.MAX_VALUE;
}
else
{
boolean visited[] = new boolean[label+1];
int j,i;//eliminate loops and find roots
for(i=1; labels[i] != Integer.MAX_VALUE ; i++)
{
if( labels[i] != Integer.MIN_VALUE )
{
for (int q = 1; q < visited.length; q++)
visited[q] = false;
visited[i] = true;
j = i;
while (labels[j] != Integer.MIN_VALUE)
{
j = labels[j];
if (visited[j])
{
labels[i] = Integer.MIN_VALUE;
break;
}
else
visited[j] = true;
}
labels[i] = j;
}
else
35
8/2/2019 Documentation_Vision Based Game Development Using Human
36/70
{
cluster++;
clusters[cluster] = i;
}
}
for (i = 1; labels[i] != Integer.MAX_VALUE ; i++)
if (labels[i] == Integer.MIN_VALUE)
labels[i] = i;
clusters[cluster + 1] = Integer.MAX_VALUE;
}
}
/////////////////////////////////////////
private int[][] findClustersCenters(int clusters[],double limit,int width,int height)
{
if( cluster == 1 ) //one cluster found
{
int centerX = 0;
int centerY = 0;
int counter = 0;
for (int y = 0; y < height; y++)
{
for (int x = 0; x < width; x++)
{
if (clustersMembers[x][y] != 0)
{
centerX += x;
centerY += y;
counter++;
}
}
}
if( counter > limit ) //if cluster size is larger than the threshold
{
36
8/2/2019 Documentation_Vision Based Game Development Using Human
37/70
int centers[][] = new int[cluster+2][2];
centers[1][0] = (int)centerX / counter;
centers[1][1] = (int)centerY / counter;
centers[cluster+1][0] = Integer.MAX_VALUE;
return centers;
}
else
return null;
}
else
{
//more than one clusterlong[] centerX = new long[cluster + 2];
long[] centerY = new long[cluster + 2];
counters = new int[cluster + 2];
indecies = new int[label + 2];
int index, clusterArea;
for (int i = 1; clusters[i] != Integer.MAX_VALUE; i++)
indecies[clusters[i]] = i;
Vector finalClusters = new Vector();
for (int y = 0; y < height; y++)
{
for (int x = 0; x < width; x++)
{
if (clustersMembers[x][y] != 0)
{
index = indecies[labels[clustersMembers[x][y]]];
centerX[index] += x;
centerY[index] += y;
counters[index]++;
}
}
}
for (int i = 1; clusters[i] != Integer.MAX_VALUE; i++)
37
8/2/2019 Documentation_Vision Based Game Development Using Human
38/70
{
clusterArea = counters[indecies[clusters[i]]];
if (clusterArea > limit)
finalClusters.add(new Integer(clusters[i]));
else
cluster--;
}
if( cluster != 0 )
{
for (int i = 0; i < finalClusters.size(); i++)
clusters[i + 1] = ( (Integer) (finalClusters.get(i))).intValue();
clusters[cluster + 1] = Integer.MAX_VALUE;int centers[][] = new int[cluster + 2][2];
for (int i = 1; clusters[i] != Integer.MAX_VALUE; i++)
{
index = indecies[clusters[i]];
centers[i][0] = (int) (centerX[index] / counters[index]);
centers[i][1] = (int) (centerY[index] / counters[index]);
}
centers[cluster + 1][0] = Integer.MAX_VALUE;
return centers;
}
else
return null;
}
}
//////////////////////////////////////////////////
private Point oneClusterCenter(int centerX,int centerY,int counter,int start
,int end,int width,double limit)
{
for (int y = start; y < end; y++) //look in the specified region
{
38
8/2/2019 Documentation_Vision Based Game Development Using Human
39/70
for (int x = 0; x < width; x++)
{
if (clustersMembers[x][y] != 0)
{
centerX += x;
centerY += y;
counter++;
}
}
}
if (counter > (double) limit / 2d)
{Point center = new Point();
center.setLocation( (int) (centerX / counter), (int) (centerY / counter));
return center;
}
else
return null;
}
//////////////////////////////////////////////////
private Point multipleClustersCenter(int clusters[],int start,int end,int width,double limit
,int grayPixels[],int x0,int y0,int fWidth)
{
long[] centerX = new long[cluster + 2];
long[] centerY = new long[cluster + 2];
long[] colors = new long[cluster + 2];
int[] counter = new int[cluster + 2];
int[] indecies = new int[label + 2];
int index = 0, clusterArea, darkestPixel = 256, pixelColor, darkestX = 0, darkestY = 0, ind;
for (int i = 1; clusters[i] != Integer.MAX_VALUE; i++)
indecies[clusters[i]] = i;
for (int y = start; y < end; y++)
{
39
8/2/2019 Documentation_Vision Based Game Development Using Human
40/70
for (int x = 0; x < width; x++)
{
if (clustersMembers[x][y] != 0)
{
index = indecies[labels[clustersMembers[x][y]]];
centerX[index] += x;
centerY[index] += y;
counter[index]++;
pixelColor = grayPixels[ (y + y0) * fWidth + (x + x0)];
colors[index] += pixelColor;
if (pixelColor < darkestPixel)
{darkestPixel = pixelColor;
darkestX = x;
darkestY = y;
}
}
}
}
double max = Double.NEGATIVE_INFINITY, ratio, dist = 0;
int cX, cY;
index = Integer.MIN_VALUE;
for (int i = 1; clusters[i] != Integer.MAX_VALUE; i++)
{
clusterArea = counter[indecies[clusters[i]]];
if (clusterArea > (double) limit / 2d)
{
ind = indecies[clusters[i]];
cX = (int) (centerX[ind] / counter[ind]);
cY = (int) (centerY[ind] / counter[ind]);
dist = Math.sqrt(Math.pow( (cX - darkestX), 2) +
Math.pow( (cY - darkestY), 2));
ratio = clusterArea / (colors[ind] * dist); //look for the largest,darkest,and closest to the
darkest pixel, cluster
if (ratio > max)
40
8/2/2019 Documentation_Vision Based Game Development Using Human
41/70
{
max = ratio;
index = ind;
}
}
}
if (index != Integer.MIN_VALUE)
{
Point center = new Point();
center.setLocation( (int) (centerX[index] / counter[index]),
(int) (centerY[index] / counter[index]));
return center;}
else
return null;
}
//////////////////////////////////////////////////
public Point findPupilsClustersCenters(int x0,int y0,int clusters[],double limit,int width,int
height,int grayPixels[],int fWidth)
{
if( cluster == 1 ) //one cluster found
{
int centerX = 0;
int centerY = 0;
int counter = 0; //look in the lower half
Point center = oneClusterCenter(centerX,centerY,counter,(int)height/2,height,width,limit);
if( center != null )
return center;
else
{
centerX = 0;
centerY = 0;
counter = 0; //look in the upper half
41
8/2/2019 Documentation_Vision Based Game Development Using Human
42/70
return oneClusterCenter(centerX,centerY,counter,0,(int)height/2,width,limit);
}
}
else
{ //look in the lower half
Point center = multipleClustersCenter(clusters,
(int)height/2,height,width,limit,grayPixels,x0,y0,fWidth);
if( center != null )
return center;
else //look in the upper half
return multipleClustersCenter(clusters,0,
(int)height/2,width,limit,grayPixels,x0,y0,fWidth);}
}
//////////////////////////////////////////////////
public int[][] processClusters(double limit,int width,int height)
{
findRootLabels();
return findClustersCenters(clusters,limit,width,height);
}
public int getCluster() {
return cluster;
}
public int getLabel() {
return label;
}
///////////////////////////////////////////////////
}
42
8/2/2019 Documentation_Vision Based Game Development Using Human
43/70
FaceDetector.java
import java.util.*;
import java.awt.*;
import svm.*;
public class FaceDetector
{
final int fWidth = 320, fHeight = 240;
int[] labels, start, binaryPixels;
public int[] grayPixels,pixels;
int[][] s, ii, clustersMembers, eyes;
int faces, cluster;
svm_model model;
svm_node nodes[];
//////////////////////////////
public FaceDetector(svm_model model)
{
pixels = new int[fWidth * fHeight];
grayPixels = new int[fWidth * fHeight];
binaryPixels = new int[fWidth * fHeight];
s = new int[fWidth][fHeight];
ii = new int[fWidth][fHeight];
clustersMembers = new int[fWidth][fHeight];
labels = new int[fWidth * fHeight];
nodes = new svm_node[735];
this.model = model;
start = new int[model.nr_class];
start[0] = 0;
for (int i = 1; i < model.nr_class; i++)
start[i] = start[i - 1] + model.nSV[i - 1];
43
8/2/2019 Documentation_Vision Based Game Development Using Human
44/70
}
////////////////////////////////////////
public int[] detectAllFaces(Image image)
{
//Initializing
pixels = ImageProcessing.extractPixels(image,0,0,fWidth, fHeight, pixels);
grayPixels = ImageProcessing.toGrayscale(pixels, grayPixels);
binaryPixels = ImageProcessing.toBinary(grayPixels,binaryPixels,128);
ii = ImageProcessing.calculateIntegralImage(fWidth, fHeight, grayPixels, s, ii);
faces = 0;int coords[];
//Apply biggest filter first
// SSRFilter filter1 = new SSRFilter(120,72, ii);
// coords = detectFaces(filter1);
//If no faces were found with the previous filter apply a smaller one, and so on...
// if (faces == 0)
{
SSRFilter filter2 = new SSRFilter(84, 54, ii);
coords = detectFaces(filter2);
}
if (faces == 0)
{
SSRFilter filter3 = new SSRFilter(60, 36, ii);
coords = detectFaces(filter3);
}
return coords;
}
////////////////////////////////////////
private int[] detectFaces(SSRFilter filter)
44
8/2/2019 Documentation_Vision Based Game Development Using Human
45/70
{
//Detect faces with the given filter
//If the pixel is a face candidate find it's cluster
int xCor, yCor, label = 0;
for (int i = 0; i < fWidth; i++)
for (int j = 0; j < fHeight; j++)
clustersMembers[i][j] = 0;
ConnectedComponents CC = new ConnectedComponents(labels, clustersMembers);
for (int y = 0; y < fHeight - filter.getHeight(); y++)
for (int x = 0; x < fWidth - filter.getWidth(); x++){
xCor = x + 1 + (filter.getWidth() / 2); //+1 : transition error in sectors calculation
yCor = y + 1 + (filter.getHeight() / 2);
if (filter.foundFaceCandidate(x, y) &&
ImageProcessing.isSkinPixel(pixels[yCor * fWidth + xCor]))
label = CC.findClustersLabels(xCor, yCor,fWidth);
}
//If there are face candidates
if (label != 0)
{
//Find root labels,and their centers
int centers[][] = CC.processClusters(filter.getArea()/48d,fWidth,fHeight);
//If there are valid clusters
if (centers != null)
{
//Find left/right pupils for each of the candidates
cluster = CC.getCluster();
int counter = findPupilsCandidates(filter,centers);
//if found some eyes
if( counter != -1 )
45
8/2/2019 Documentation_Vision Based Game Development Using Human
46/70
{
//Load templates to the array in order to classify them
Vector templates = loadEyesTemplates(counter);
//Classiffying templates
double cResults[] = classify(templates);
//Multiply classification results by clusters' areas
if( cluster > 1 )
cResults = multiplyByArea(CC.indecies,CC.clusters,CC.counters,cResults);
//Find the template with the highest scoreint face[] = findBestEyesTemplate(cResults);
//Find nose tip
if( face != null )
{
Point noseTip = findNoseTip(face);
if( noseTip != null )
{
int coordinates[] = new int[6];
coordinates[0] = face[0];
coordinates[1] = face[1];
coordinates[2] = face[2];
coordinates[3] = face[3];
coordinates[4] = (int)noseTip.getX();
coordinates[5] = (int)noseTip.getY();
return coordinates;
}
}
}
}
}
return null;
}
46
8/2/2019 Documentation_Vision Based Game Development Using Human
47/70
//////////////////////////////////////////////
private Point findPupil(int x0, int y0,int width,int height,int labels[], int pupilsMembers[]
[],int limit)
{
int label = 0; //binarize the sector and look for the appropriate cluster
for (int y = 0; y < height; y++)
for (int x = 0; x < width; x++)
pupilsMembers[x][y] = 0;
ConnectedComponents CC = new ConnectedComponents(labels,pupilsMembers);
for (int y = 0; y < height; y++)for (int x = 0; x < width; x++)
if( binaryPixels[ (y + y0) * fWidth + (x + x0)] == 1 )
label = CC.findClustersLabels(x, y, width);
if( label != 0 )
{
CC.findRootLabels();
return
CC.findPupilsClustersCenters(x0,y0,CC.clusters,limit,width,height,grayPixels,fWidth);
}
else
return null;
}
/////////////////////////////////////////////
private int findPupilsCandidates(SSRFilter filter,int centers[][])
{
eyes = new int[cluster][5];
int width = filter.getWidth()/3;
int height = filter.getHeight()/2;
int labels[] = new int[width*height];
int pupilsMembers[][] = new int[width][height];
int limit = (int)filter.getArea()/(144*6);
47
8/2/2019 Documentation_Vision Based Game Development Using Human
48/70
int halfW, halfH, sixthW, counter = -1;
halfW = filter.getWidth() / 2;
halfH = filter.getHeight() / 2;
sixthW = filter.getWidth() / 6;
Point lp1,rp1;
int x, y;
for (int i = 1; centers[i][0] != Integer.MAX_VALUE; i++)//pass all clusters
{
x = centers[i][0];
y = centers[i][1];
//find left pupil in sector s1
lp1 = findPupil(x-halfW,y-halfH,width,height,labels,pupilsMembers,limit);if( lp1 == null )
continue;
//find right pupil in sector s3
rp1 = findPupil(x+sixthW,y-halfH,width,height,labels,pupilsMembers,limit);
if( rp1 == null )
continue;
counter++;
eyes[counter][0] = (int)lp1.getX()+x-halfW;
eyes[counter][1] = (int)lp1.getY()+y-halfH;
eyes[counter][2] = (int)rp1.getX()+x+sixthW;
eyes[counter][3] = (int)rp1.getY()+y-halfH;
eyes[counter][4] = i;
}
return counter;
}
//////////////////////////////////////////
private svm_node[] extractEyesTemplate(int x0,int y0,int x1,int y1)
{
int xLen,yLen,sX,sY,cX,cY,oX,oY;
48
8/2/2019 Documentation_Vision Based Game Development Using Human
49/70
double xInc,yInc,nY,scale;
svm_node node;
svm_node template[] = new svm_node[735];
xLen = x1-x0;
yLen = y1-y0;
xInc = (double)xLen / 23d;
yInc = (double)yLen / 23d;
scale = Math.sqrt(Math.pow(xLen,2)+Math.pow(yLen,2))/23d;
oX = x0 - (int)(6*xInc) + (int)(8*yInc);
nY = (yLen != 0 ? (8*yInc*xLen) / yLen : 8*scale);
oY = y0-(int)nY;
sX = oX;sY = oY; //derotate the template to a horizontal position
for (int y = 0; y < 21; y++)
{
cX = sX;
cY = sY;
for (int x = 0; x < 35; x++)
{
node = new svm_node();
node.index = y * 35 + x + 1;
if ( (cX >= 0) && (cX < fWidth) && (cY < fHeight) && (cY >= 0) )
node.value = grayPixels[cY * fWidth + cX] / 255d;
else
if( y == 0 )
node.value = 128d/255d;
else
node.value = template[ (y - 1) * 35 + x].value;
template[y * 35 + x] = node;
cX = sX+(int)((x+1)*xInc);
cY = sY+(int)((x+1)*yInc);
}
sX = oX-(int)((y+1)*yInc);
sY = oY+(int)((y+1)*xInc);
}
49
8/2/2019 Documentation_Vision Based Game Development Using Human
50/70
return template;
}
//////////////////////////////////////////
private Vector loadEyesTemplates(int counter)
{
svm_node[] template;
Vector templates = new Vector();
//Extract the templates and write them in the 'LibSVM' file format
for (int i = 0; i
8/2/2019 Documentation_Vision Based Game Development Using Human
51/70
//////////////////////////////////////////////////
private double[] multiplyByArea(int indecies[],int clusters[],int counters[],double cResults[])
{
for (int i = 0; i < cResults.length; i++)
if( cResults[i] > 0 )
cResults[i] *= counters[indecies[clusters[eyes[i][4]]]];
return cResults;
}
//////////////////////////////////////////////////
private int[] findBestEyesTemplate(double cResults[])
{
double max = 0d;
int index = 0;
for (int i=0 ; i< cResults.length ; i++)
if( cResults[i] > max )
{
max = cResults[i];
index = i;
}
if( max>0 )
{
faces++;
int face[] = new int[4];
face[0] = eyes[index][0];
face[1] = eyes[index][1];
face[2] = eyes[index][2];
face[3] = eyes[index][3];
return face;
}
else
51
8/2/2019 Documentation_Vision Based Game Development Using Human
52/70
return null;
}
//////////////////////////////////////////////////
private Point[] findNoseBridge(int length,int ii[][])
{
int val,max,nbcx = 0,counter = 0;
SSRFilter filter = new SSRFilter(length/2,1,ii);
Point candidates[] = new Point[length];
for(int y=0 ; y < length ; y++){
max = Integer.MIN_VALUE;
for(int x=0 ; x < length-filter.getWidth() ; x++)
{
val = filter.findNoseBridgeCandidate(x,y);
if( val > max )
{
max = val;
nbcx = x + 1 + (filter.getWidth() / 2);
}
}
if( max != Integer.MIN_VALUE )
{
counter++;
candidates[y] = new Point(nbcx, max);
}
}
if( counter > (length/5) )
return candidates;
else
return null;
}
52
8/2/2019 Documentation_Vision Based Game Development Using Human
53/70
////////////////////////////////////////////////
private int findNextCandidate(int i,Point candidates[])
{
while( (candidates[i] == null) && (i < candidates.length-1) )
{
i++;
}
return i;
}
////////////////////////////////////////////////
private int[] calculateGradiants(Point candidates[],int length)
{
int gradiants[] = new int[length],i=0,ind1=0,ind2=0;
for (int q = 0; q < gradiants.length; q++)
gradiants[q] = Integer.MAX_VALUE;
Point c1,c2;
while( true )
{
ind1 = findNextCandidate(ind2,candidates);
c1 = candidates[ind1];
if( c1 != null )
{
if (ind1 + 1 < length)
{
ind2 = findNextCandidate(ind1 + 1, candidates);
c2 = candidates[ind2];
if (c2 != null)
{
gradiants[ind2] = (int)(c2.getY() - c1.getY());
if( ind2 == length-1 )
break;
}
53
8/2/2019 Documentation_Vision Based Game Development Using Human
54/70
else
break;
}
else
break;
}
else
break;
}
return gradiants;
}
//////////////////////////////////////////////////
private Point findNoseTip(int face[])
{
int x1,y1,x2,y2,xLen,yLen,length,step,sX,sY,cX,cY;
double slope;
x1 = face[0];
y1 = face[1];
x2 = face[2];
y2 = face[3];
xLen = x2-x1;
yLen = y2-y1;
length = (int)Math.sqrt(Math.pow(xLen,2)+Math.pow(yLen,2));
step = Math.abs(yLen != 0 ? xLen/yLen : xLen);
step = ( step < 3 ? 3 : step );
slope = ( yLen < 0 ? -1 : 1);
int ROI[] = new int[length*length];
sX = x1;
sY = y1;
for (int y = 0; y < length; y++) //extract face 'Rigion Of Interest'
{
cX = sX;
cY = sY;
for (int x = 0; x < length; x++)
54
8/2/2019 Documentation_Vision Based Game Development Using Human
55/70
{
if ( (cX >= 0) && (cX < fWidth) && (cY < fHeight))
ROI[y * length + x] = grayPixels[cY * fWidth + cX];
else
ROI[y * length + x] = ROI[(y-1)*length+x];
cX++;
if ( (x + 1) % step == 0)
cY = (int) (cY + slope);
}
if ( (y + 1) % step == 0)
sX = (int) (sX - slope);
sY++;}
int ii[][] = new int[length][length];
int s[][] = new int[length][length];
ii = ImageProcessing.calculateIntegralImage(length,length,ROI,s,ii);
Point candidates[] = findNoseBridge(length,ii); //find nose bridge candidates
if( candidates != null )
{
int gradiants[] = calculateGradiants(candidates, length);
int uMin1 = Integer.MAX_VALUE,uMin2 = Integer.MAX_VALUE,
lMin1 = Integer.MAX_VALUE,lMin2 = Integer.MAX_VALUE;
int lMinGrad,lMinIndex,uMinGrad,uMinIndex,minGrad,minIndex;
int uInd1=0,uInd2=0,lInd1=0,lInd2=0,gLength;
gLength = gradiants.length;
for(int i=0 ; i
8/2/2019 Documentation_Vision Based Game Development Using Human
56/70
uInd2 = i;
}
for(int i=gLength/2 ; i= 0.5 )
{
uMinGrad = uMin2;
uMinIndex = uInd2;
}
else
{
uMinGrad = uMin1;
uMinIndex = uInd1;
}
if( (double)uMinGrad/(double)lMinGrad >= 0.5 )
56
8/2/2019 Documentation_Vision Based Game Development Using Human
57/70
{
minGrad = lMinGrad;
minIndex = lMinIndex;
}
else
{
minGrad = uMinGrad;
minIndex = uMinIndex;
}
int start;
if( minIndex >= gLength/2)
{if (minIndex < 3 * gLength / 4)
start = gLength / 2;
else
start = 3 * gLength / 4;
}
else
start = 0;
int max = 0,index = 0;
for( int i=start ; i max) && (gradiants[i] != Integer.MAX_VALUE) ) //gradiant
above it ( nose tip )
{
max = gradiants[i];
index = i;
}
if( candidates[index] == null )
return null;
Point noseTip = new Point((int)candidates[index].getX(),index);
slope = (double)yLen / (double)xLen;
double angle = Math.atan(slope); //rotate ROI to it's original state
double x = Math.cos(angle)*noseTip.getX()-Math.sin(angle)*noseTip.getY();
double y = Math.sin(angle)*noseTip.getX()+Math.cos(angle)*noseTip.getY();
57
8/2/2019 Documentation_Vision Based Game Development Using Human
58/70
x += face[0];
y += face[1];
noseTip.setLocation(x,y);
return noseTip;
}
return null;
}
Chapter 7
TESTING
7.1 Introduction to testing
Testing is a process used to help identify the correctness, completeness and quality of
developed computer software. With that in mind, testing can never completely establish the
correctness of computer software it just find out the faults whatever we did in our project.
Testing helps is verifying and Validating if the Software is working as it is intended
to be working. This involves using Static and Dynamic methodologies to Test the application.
Software
spends more time being maintained than being developed.
Things that
were obvious in development are easily obscured.
Fixing bugs iseasier than finding them and making sure nothing else breaks.
The real cost
of software is programmer time and effort, not length of code, licenses or required
hardware.
7.2 Test Cases
58
8/2/2019 Documentation_Vision Based Game Development Using Human
59/70
Test cases are a well define Scenario to perform a task to validate the Inputs and outputs. Test
cases are a sequence of steps to test the correct behavior of a functionality/feature of an
application.
The test cases are given below.
Table 6.2 Sample Test Cases to check the Functionality of the Application
59
8/2/2019 Documentation_Vision Based Game Development Using Human
60/70
Scenario ID Test case
Description
Steps Expected
Results
Actual
Results
LIVE VIDEO
CAPTURE
Test whether the
live video is
captured in the
captured image
segment.
1.Installing the
webcam
2.Initializing the
JMF
1.Live video
is detected
2. All the
connected
devices will
be detected.
1. Live video
is detected
2. All the
connected
devices will
be detected
FACE
DETECTION
Test whether the
face is detected
1. Click the
DETECT FACE
button.
1. Face is
detected
2.Detecting
the facial
features
1. Face is
detected
2.Detecting
the facial
features(nose
Eyebrows,
Eyes)
FACE
TRACKING
To track the face
movement while
changing the
location of face
1. Movement of
capture figure is
necessary
1. The facial
features are
to be
detected
while
changing the
location of
the face
1The facial
features are
not exactly
detected
while
changing the
location of
the face
ENABLE THE
INTERFACE
To test 1. Click
ENABLE
INTERFACE
1. While click
the button, the
mouse is
replaced by our
facial features
1. While click
the button, the
mouse is
replaced by our
facial features
60
8/2/2019 Documentation_Vision Based Game Development Using Human
61/70
Chapter 8
CONCLUSION AND FUTURE ENHANCEMENTS
8.1 Conclusion
We have successfully implemented the project. Thus we have developed a project capable of
replacing mouse with facial features to interact with computer.
The main advantage of this project is that only a simple webcam with moderate
resolution is sufficient to capture the live video. People with hand disabilities can use this
project effectively.
The disadvantage of this project is that involuntary eye blinks also trigger mouse clicks.
Also it may lead to strain in the eyes due to continuous blink.
8.2 Future Enhancement
Involuntary eye blinks can be detected and they can be prevented from firing the events.
The sensitivity of nose movement is very high that even a small reaction induces
greater mouse movement.
61
8/2/2019 Documentation_Vision Based Game Development Using Human
62/70
Appendix A
SNAPSHOTS
A.1 Login GUI
62
8/2/2019 Documentation_Vision Based Game Development Using Human
63/70
A.2 Detecting the Face
63
8/2/2019 Documentation_Vision Based Game Development Using Human
64/70
A.3 Face Tracking
64
8/2/2019 Documentation_Vision Based Game Development Using Human
65/70
A.4 Enabled Visage
65
8/2/2019 Documentation_Vision Based Game Development Using Human
66/70
Appendix B
ACRONYMS AND ABBREVATIONS
66
8/2/2019 Documentation_Vision Based Game Development Using Human
67/70
BTE Between the eyes
HCI Human Computer Science
JMF Java Media Framework
SSD Squared Difference
SSR Six-Segmented Rectangular Filter
NBP Nose Bridge Point
GUI Graphical User Interface
AWT Abstract Windowing Toolkit
JAR Java Archive File
Appendix c
USER MANNUAL
67
8/2/2019 Documentation_Vision Based Game Development Using Human
68/70
Installing the Java Standard Edition Development Kit (JDK) 5
Our application is developed on the java standard edition development kit (JDK) 5. You can
download the latest version of JDK5 and its documentation from
Java.sun.com/javase/5/download.jsp
After downloading the JDK installer, double click the installer program to begin
installing the JDK. We recommended that you accept all the default installation options. If you
change the default installation directory, be sure to write down the exact name and location of
the directory you choose, as you will need this information later in the installation process on
Windows JDK is placed in the following directory by default:
C:\Program Files\Java\jdk.5.0
Installing the Java Media Framework 2.1.1e
For the installation of the JMF first we have to download the JMFs, version 2.1.1e, installation
file and after downloading the JMF installer, double click the installer program to begin
installing the JMF. We recommended that you accept all the default installation options.
Installing the web camera drivers
To install the driver, we need the setup file of the web camera drivers. We can download or we
can use the installation CD which contains the setup files. And then, double click the installer
program to begin installing the drivers. We recommended that you accept all the default
installation options.
Running the application
68
8/2/2019 Documentation_Vision Based Game Development Using Human
69/70
To run the application user need to save the application JAR file in our computers secondary
memory, and then we have to click on the JAR file, then we get login GUI. Through the GUI
the user can operate the application. The GUI, having option,
Detect the Face- this key is used to tracking the face with the help of users eye blink and nosetip.
Enable Visage- This key is used to enable the application for the user who got tracked. After
enabling the visage the user is ready to control the computer.
Stop Tracking- This key is used to stop the tracking. This option is useful when the user want
to replace them self to another user before enabling visage.
Refresh Preview- This button is used to refresh the preview.
BIBLIOGRAPHY
69
8/2/2019 Documentation_Vision Based Game Development Using Human
70/70
REFERENCES
[1] E.Hjelmas and B.K.Low(2001), Face Detection: A survey, Computer Vision and Image
Understanding.
[2] S.Kawato and N. Tetsutani( January 2002), Real-time detection of Between-the-Eyes with
a Circle Frequency Filter: The 5th Asian Conference on Computer Vision,Melbourne,
Australia.
[3] S.Kawato and J.Ohya,(October 2000) Two-step Approach for Real- Time Eye Tracking
with a New Filtering Technique: IEEE Int. Conf. on Systems, Man & Cybernetics, Nashville,
Tennessee, USA.
[4] P.Viola and M.Jones(2001), Rapid object Detection using a Boosted Cascade of Simple
Features, Proc. of IEEE Conf.CVRP.
[5] J.Yang and A.Waibel(1996), A real-time face tracker, Proc.3rd IEEE Workshop on
Application of Computer Vision.
[6] AT&T Laboratories Cambridge, The ORL face
database,http://www.uk.research.att.com/facedatabase.html
[7] http://s.i-techonline.com/Book/Humanoid-Robots-New-Developments/ISBN978-3-902613-
00-4hrnd14.pdf
URLs:
WWW.WIKIPEDIA.ORG
WWW.SOURCEFORGE.NET
WWW.HOWSTUFFWORKS.COM
http://www.us.design-reuse.com/exit?url=http://www.uk.research.att.com/facedatabase.htmlhttp://www.us.design-reuse.com/exit?url=http://www.uk.research.att.com/facedatabase.htmlhttp://www.wikipedia.org/http://www.sourceforge.net/http://www.howstuffworks.com/http://www.wikipedia.org/http://www.sourceforge.net/http://www.howstuffworks.com/http://www.sun.java.com/http://www.us.design-reuse.com/exit?url=http://www.uk.research.att.com/facedatabase.htmlTop Related