Documentation_Vision Based Game Development Using Human

8/2/2019 Documentation_Vision Based Game Development Using Human

1/70

Abstract

A Human Computer Interface (HCI) System for playing games is designed here for more

natural communication with the machines. The system presented here is a vision-based systemfor detection of long voluntary eye blinks and interpretation of blink patterns for

communication between man and machine. This system replaces the mouse with the human

face as a new way to interact with the computer. Facial features (nose tip and eyes) are detected

and tracked in real-time to use their actions as mouse events. The coordinates and movement of

the nose tip in the live video feed are translated to become the coordinates and movement of the

mouse pointer on the application. The left/right eye blinks fire left/right mouse click events.

The system works with inexpensive USB cameras and runs at a frame rate of 30 frames per

second.

1


2/70

Chapter 1

INTRODUCTION

1.1 Purpose of the project

In the past few years high technology has become more progressed, and less expensive. With

the availability of high speed processors and inexpensive webcams, more and more people

have become interested in real-time applications that involve image processing. One of the

promising fields in artificial intelligence is human computer interaction which aims to use

human features (e.g. face, hands) to interact with the computer. One way to achieve that is to

capture the desired feature with a webcam and monitor its action in order to translate it to some

events that communicate with the computer.

The current evaluation of computer technologies has enhanced various applications in

human-computer interface. Face and gesture recognition is a part of this field, which can be

applied in various application such as in robotic, security system, drivers monitor, image

processing.

Since human face is a dynamic object and has a high degree of variability. Face detection

can be classified as two categories feature-based approach and image-based approach. The

techniques in the first category makes used of apparent properties of face such as geometry,

skin color, and motion. Even feature-based technique can achieve high speed in face detection,

but it also has problem in poor reliability under lighting condition. For second category, the

image-based approach takes advantage of current advance in pattern recognition theory. Most

of the image-based approach applies a window scanning technique for detecting face, whichrequires large computation. Therefore, by using only image-based approach is not suitable

enough in real-time application.

In order to achieve high speed and reliable face detection system, we propose the

method combine both feature-based and image-based approach to detect the point between the

eyes by using SSR[1].The proposed SSR filter, which is the rectangle divided into 6 segments,

operates by using the concept of bright-dark relation around BTE [2] area. We select BTE [2]

as face representative because it is common to most people and easy to find for wide range of

2


3/70

face orientation. BTE [2] has dark part (eyes and eyebrows) on both sides, and has comparably

bright part on upper side (forehead), and lower side (nose and cheekbone).

We use an intermediate representation of image called integral image to calculate sums

of pixel values in each segment of SSR [1] filter. Firstly, SSR [1] filter is scanned on the image

and the average gray level of each segment is calculated from integral image. Then, the bright-

dark relations between each segment are tested to see whether its center can be a candidate

point for BTE [2]. Next, the camera is used to find the distance information and the suitable

BTE [2] template size. Then, the BTE [2] candidates are evaluated by using a template of BTE

[2]. Finally the true BTE [2], nose tip can be detected.

Using the co-ordinate of mouse tip as pointer and eye blink fires the left/right mouse clickevents.

1.2 Objective

Our system is capable of replacing the traditional mouse with the human

face a new way to interact with the computer.

Facial features (nose tip and eyes) are used to interact with computer.

The nose tip acts as a mouse pointer and the left/right eye blinks fire

left/right mouse click events.

3


4/70

Chapter 2

LITERATURE REVIEW

2.1 Existing System

HCI (Human Computer Interface) aims to use human features (e.g. face, hands) to interact with

the computer. One way to achieve that is to capture the desired feature with a webcam and

monitor its action in order to translate it to some events that communicate with the computer.

Traditional Mouse

Wireless Mouse

Roller bar Mouse

2.1.1 Traditional Mouse

In computing, a mouse (plural mice or mouses) functions as a pointing device by detecting

two-dimensional motion relative to its supporting surface. Physically, a mouse consists of a

small case, held under one of the user's hands, with one or more buttons. It sometimes features

other elements, such a "wheels", which allow the user to perform various system-dependent

operations, or extra buttons or features can add more control or dimensional input. The mouse'smotion typically translates into the motion of a pointer on a display, which allows for fine

control of a Graphical User Interface.The name mouse originated at the Stanford Research Institute derives from the

resemblance of early models (which had a cord attached to the rear part of the device,

suggesting the idea of a tail) to the common mouse.

Fig 2.1Wired mouse

2.1.2 Wireless Mouse

4


5/70

A Wireless Mouse is a computer mouse that communicates with the CPU via RF wireless

ID technology instead of via a plug-in cord.

Fig 2.2 Wireless mouse

Technical features of the mouse are

Touch-sensitive top shell

360 degree enabled clickable scroll ball

Force-sensing side "squeeze" buttons

Optical tracking in wired version or laser tracking in wireless version

Compatible with Macintosh and PCs

Programmable functions for the four buttons

Easter egg: when held parallel above a surface, the shape of the light

projected from the wired USB mouse resembles the head of a mouse

2.1.3 Roller Mouse

Roller bars are one of the more esoteric computers pointing devices, consisting essentially of a

round, rubber coated bar situated between the computer keyboard and the user, who

manipulates the screen cursor through a combination of rolling the bar on its axis and sliding it

sideways in its housing.

The operation is a lot more intuitive than it sounds, and the theory behind it is that

roller bars reduce physical stress, especially on the elbows and shoulders, by eliminating the

reaching necessary when using any movable mouse - conventional or ergonomic.

5


6/70

Contour Design's Roller mouse roller bar station is designed to be used with standard

computer keyboards that have a rectangular form factor and a straight edge on the user side.

Fig 2.3 Roller bar mouse

The Roller Mouses roller bar is contained in a full-width housing that incorporates a

console of click buttons and a scroll wheel flanked by padded wrist rests. An apron extends

from the control housing for the keyboard to rest on.

The Roller Mouse controls the cursor without the need to grip or reach for a mouse, and

requires minimal space. Button controls are centrally located beneath the space bar. The Roller

Mouse is also a solution for workspaces where there is no room for a conventional mouse.

The roller bar can be manipulated using either hand or whatever combination of

thumb, fingers, or palm control is most comfortable. This facility for spreading mousing stress

between both hands and various digits is a key element in reducing the overall stress on the

user's body.

Clicking may also be executed in several ways, either with the three buttons andclickable scroll wheel or by pressing down on the roller bar itself, which is clickable as well.

Button assignments re programmable, with defaults as follows.

6


7/70

Fig 2.4 Working of roller bar

Disadvantage of Roller Mouse

Clickable roller bar: primary or left button

Left click button: primary or left button Middle click button: preprogrammed for double click

Right click button: secondary or right button (i.e.: activates contextual

menus)

Scroll wheel button: auxiliary or middle button

We interchange hand from keyboard to mouse while working in

computer

It takes large time to work

Hand disabilities persons are unable to work properly

2.2 Proposed System

Our system is capable of replacing the traditional mouse with the human face - a new way to

interact with the computer. Facial features (nose tip and eyes) are used to interact with

computer. The nose tip acts as a mouse pointer and the left/right eye blinks fire left/right mouse

click events .Only external device that the user needs is a webcam that feeds the program with

the video stream .The people having hand disabilities are able to work.

Chapter 3

7


8/70

SYSTEM REQUIREMENTS

3.1 Introduction

The Java Media Framework (JMF) is a recent API for Java dealing with real-time multimedia

presentation and effects processing. JMF handles time-based media, media which changes

with respect to time. Examples of this are video from a television source, audio from a raw-

audio format file and animations. The beta JMF 2.0 specification will be used for this report,

as they currently reflect the features that will appear in the final version.

3.2 Stages

The JMF architecture is organized into three stages:

Fig 3.1 JMF Architecture

During the input stage, data is read from a source and passed in buffers to the

processing stage. The input stage may consist of reading data from a local capture device (such

as a webcam or TV capture card), a file on disk or stream from the network.

The processing stage consists of a number of codecs and effects designed to modify the

data stream to one suitable for output. These codecs may perform functions such as

compressing or decompressing the audio to a different format, adding a watermark of some

kind, cleaning up noise or applying an effect to the stream (such as echo to the audio).

8


9/70

Once the processing stage has applied its transformations to the stream, it passes the

information to the output stage. The output stage may take the stream and pass it to a file on

disk, output it to the local video display or transmit it over the network.

3.3 Component Architecture

JMF is built around component architecture. The components are organized into a number of

main categories:

Media handlers

Data sources

Codecs/Effects

Renderers

Mux/Demuxes

3.3.1 Media Handlers

Media Handlers are registered for each type of file that JMF must be able to handle. To

support new file formats, a new Media Handler can be created.

3.3.2 Data Source

A Data Source handler manages source streams from various inputs. These can be for network

protocols, such as http or ftp, or for simple input from disk.

3.3.3 Codecs/Effects

Codecs and Effects are components that take an input stream, apply a transformation to it and

output it. Codecs may have different input and output formats, while Effects are simple

transformations of a single input format to an output stream of the same format.

3.3.4 Renderers

A renderer is similar to a Codec, but the final output is somewhere other than another stream.

A VideoRenderer outputs the final data to the screen, but another kind of renderer could output

to different hardware, such as a TV out card.

9


10/70

3.3.5 Muxs / Demuxes

Multiplexers and Demultiplexers are used to combine multiple streams into a single stream or

vice-versa, respectively. They are useful for creating and reading a package of audio and

video for saving to disk as a single file, or transmitting over a network.

3.4 System Specification

The Hardware and software required for implementing this project are listed below.

3.4.1 Hardware Requirement (Minimum Requirement)

Processor Pentium III 700MHz

CPU 500 MHz

RAM 128 MB

Monitor 17 Samsung color monitor

Hard Disk 40 GB

Keyboard Standard Keyboard with 104 Keys

Mouse Serial mouse

Camera Web Camera

3.4.2 Software Requirement

Tools Java 5.0, Java Media Framework 2.1.1e

Platform Windows 98/2000, Windows XP

10


11/70

Chapter 4

DESIGN

4.1 System Designing

The skeleton of the entire process is prepared in this phase

The scheduling and interactivity of the system is completed in designing

The work structure including look and feel is generated for the system

4.1.1 Input Design

Input design is the process of converting user-originated inputs to a computer based format.

Input design of the Eye encompasses various controls such as buttons and menu items. The

refresh and stop buttons are used to refresh and stop receiving the video frames from the video

input device .The enable interface button is used to deactivate the mouse and act as a interface

for interacting with the system. The face detection button is used to detect the face candidates

(nose tip and eyes).The menu items such as file, help are used to perform normal operations

such as exit and to know the information about the project.

4.1.2 Output Design

Output design generally refers to the results generated by the system and it is a direct source of

information to the user. For many end users, output is the main reason for developing the

system and the basis on which they evaluate the usefulness of the application. The objective of

a system finds its shape in terms of the output.

The output of this project tracks and identifies the human facial features (nose tip and

eyes).

4.2 Modules

Module 1

Live Video Capture

Face Detection Find Face Candidates

Module 2

11


12/70

Extract Between The Eyes

Find Nose Tip

Module 3

Face tracking

Module 4

Front end

4.3 Block Diagram

Fig 4.1 Block diagram for eye controlled interaction

In video frame module we capture the live video, track the nose tip by using SSR filterand the ROI [4] concepts. Then BTE [2] is tracked, the sector is binarized with a certain

threshold. The cluster of the binarized sectors is found. Detecting eye brows is done by using

Squared Difference (SSD).Detect eye blink by using Motion Detection algorithm. For tracking

the eyes we use SSR [1] filter. Finally refining the nose tip is done by using nose bridge

detection with SSR filter and horizontal profile.

12


13/70

4.4 Flow Diagram

Fig 4.2 Flow Diagram for eye controlled interaction

SSR FILTER

STERO CAMERA TO FIND THE

DISTANCE INFORMATION

AVERAGE BETWEEN THE EYE

TEMPLATE MATCHING

EYE DETECTION

BETWEEN THE EYE POINT

DETECTION

CANDIDATE POINT

BETWEEN THE EYE

TEMPLATE

13


14/70

4.5 Data Flow Diagram

Fig 4.3 Data Flow Diagram 1

14


15/70



15


16/70

Chapter 5

IMPLEMENTATION

5.1 Problem Statement

In traditional mouse or in any wireless mouse, movement of the hand is necessary to select or

deselect the object. This becomes difficult for those with hand disabilities.

5.2 Project Developed

In our project we overcome this by using facial features to interact with computer i.e., nose tip

is tracked and it is used as mouse pointer and left and right blinks fire left and right

respectively. Our project consist 3 modules.

5.3 Module1

5.3.1 Live Video Capture

The first module is to capture the Live Image using the Web Cam and to detect a face in the

captured Image Segment.

5.3.2 Face Detection Module

Face Detection can be categorized as

Feature Based Method

Image Based Method

5.3.2.1 Feature Based Method

This techniques makes use of

apparent properties of face such as geometry, skin color, and motion

Lack of Robustness against head

rotation ,scaling and problem under lightning condition

5.3.2.2 Image Based Method

The image of

interest with a window that looks for faces at all scales and locations.

16


17/70

Face detection

implies pattern recognition, and achieves it with simple methods such as template

matching

Not suited for

real time application

In order to achieve the high speed and reliable face detection system, we

combine these two techniques by using Six-Segmented Rectangular filter (SSR Filter) in figure

5.1

Fig 5.1 SSR Filter

The sum of pixels in each sector is denoted as S along with the sector number. The use

of this filter will be explained in detail in the face detection algorithm.

5.3.2.3 Integral Image

In order to facilitate the use of SSR filters an intermediate image representation called integral

image will be used. In this representation the integral image at location x, y contains the sum of

pixels which are above and to the left of the pixel x, y (see fig. 5.2)

17


18/70

Fig 5.2 Integral Imageii: Integral image, i: Pixel value.

With this representation calculating the sectors of the SSR filter becomes fast and easy. No

matter how big the sector is, we will need only 3 arithmetic operations to calculate the sum of

pixels which belong to it (see fig.5.2).Sector = D B C + A (5.1)

So each SSR filter requires 6*3 operations to calculate it.

5.3.3 Find Face Candidates Module

The human face is governed by proportions that define the different sizes and distances

between facial features. We will be using these proportions in our heuristics to improve facial

features detection and tracking.

5.3.3.1 Face detection general steps

We will be using feature based face detection methods to reduce the area in which we are

looking for the face, so we can decrease the execution time. To find face candidates the SSR

[1] filter will be used in the following way:

At first we calculate the integral image by making a one pass over the video frame

using these equations:

s(x, y) = s(x, y-1) + i(x, y) (5.2)

ii(x, y) = ii(x-1, y) A+ s(x, y) (5.3)

Where s(x, y) is the cumulative row sum, s(x,-1) = 0, and ii(-1, y) =0. Figure shows an ideal

location of the SSR [1] filter, where its center is considered as a face candidate.

18


19/70

Fig5.3 Finding face candidate

(x, y) is the location of the filter (upper left corner). The plus sign is the center of the filter

which is the face candidate.

We can notice that in this ideal position the eyes fall in sectors S1 and S3, while the

nose falls in sector S5. Since the eyes and eye brows are darker than the BTE and the cheek

bones, we deduce that :

S1 < S2 && S2 > S3 (5.4)

S1 < S4 && S3 < S6 (5.5)

So in order to find face candidates we place the upper left corner (see fig.5.3) of the

SSR filter on each pixel of the image (only on pixels where the filter falls entirely inside the

bounds of the image) .For each location (x, y) we check equations (5.4,5.5); if the conditions

are fulfilled then the center of the filter will be considered as a face candidate if its a and b

values fulfill equations 5.4 and 5.5 (skin pixel thresholds).Eventually the candidates will group

in clusters.

5.4 Module 2

5.4.1 Extract BTE (Between the Eyes) Module

In order to extract BTE templates we need to locate pupils candidates. Left and right pupils

candidates will fall in sectors S1, S3. For each of the earlier mentioned sectors the following

steps are applied to find the pupils. To find the pixels that belong to a dark area (in our case the

19


20/70

pupil); the sector is binarized with a certain threshold. The clusters of the binarized sector are

found.

1. If the thresholding produces only one cluster like sector S3 (right eye) in figure 5.3, it is a

large probability that the clusters of the eye brow and the pupil has unified in one, so in this

case it is likely to look for the pupil in the lower half of the sector cause as we already

mentioned the upper half is probably the eye brow. So what we are going to do is to calculate

the area of the part of the cluster which is in the lower half of the sector, if it is larger than a

specified threshold then the center of the lower part is considered as the pupil; if not the same

will be applied to the upper half because it is possible that the cluster that was found is the

pupil cluster alone (without the eye brow cluster) and it fell in the upper half. In case of failing

to find a pupil in the upper half this sector will be Omitted and no pupil is found.

2. If there are multiple clusters like sector S1 (left eye), we need to find the cluster that is:

The largest: some clusters are the result of video noise and they are small.

The darkest: the pupil is darker from other features in the sector

The closest: to the darkest pixel of the sector: the eye pupil is black and we need to find

the cluster that contains the pupil or is the closest to it.

These criteria are applied in the lower half of the sector first, because as already

mentioned we need to avoid picking the eye brow cluster as the pupil, if the chosen cluster is

larger than a certain threshold its center will be considered as the pupil, if not, we will do the

same in the upper half, in case of failing to find a pupil in the upper half this sector will be

omitted and no pupil is found. If we didnt find a left or right pupil candidate the cluster will be

skipped and no longer considered as a face candidate.

5.4.2 Find Nose Tip Module

Now that we located the eyes, the final step is to find the nose tip. From the following figure

we can see that the blue line defines a perfect square of the pupils and outside corners of the

mouth; the nose tip should fall inside this square, so this square becomes our region of interest

in finding the nose tip.

20


21/70

Fig 5.4 Location of ROI

So the first step is to extract the ROI [4], in case the face was rotated we need to rotate

the ROI [4] back to a horizontal alignment of the eyes. The nose tip has a convex shape so it

collects more light than other features in the ROI because it is closer to the light source. Using

the previous idea we tried to locate the nose tip with intensity profiles in horizontal intensity

profiles we add vertically to each line the values of the lines that precedes it in the ROI [4], so

since that the nose bridge is brighter than the surrounding features the values should

accumulate faster at the bridge location; in other words the maximum value of the horizontal

profile gives us the x coordinate of the nose tip. In vertical intensity profiles we add

horizontally to each column the values of the columns that precedes it in the ROI [4] (see fig.

5.5); the same as in the horizontal profile, the values accumulate faster at the nose tipposition

so the maximum value gives us the y coordinate of the nose tip. From both, the horizontal and

vertical profiles we were able to locate the nose tip position, but unfortunately this method did

not give accurate results because there might be several max values in a profile that are close to

each other, and choosing the correct max value that really points out the coordinates of the nose

tip is a difficult task.

So instead of using the intensity profiles alone, we will be applying the following

method.

At first we need to locate the nose bridge and then we will find the nose tip on that

bridge. As earlier mentioned the nose bridge is brighter than surrounding features, so we will

use this criterion to locate the nose-bridge-point (NBP) on each line of the ROI.

21


22/70

We will be using an SSR filter to locate the NBP [5] candidates in each ROI [4] line.

The width of the filter is set to the half of the distance between the eyes, because from figure

5.5 we can notice that the line (nose width) is equal to the half of the line (distance between the

eyes).

5.4.2.1 SSR filter to locate nose bridge candidates

After calculating the integral image of the ROI [4], each line of it will be scanned with this

filter; we remember that the nose bridge is brighter than the regions to the left and right of it; in

other words the center of the SSR filter is considered as an NBP [5] candidate if the center

sector is brighter than the side sectors:

S2 > S1 (5.5)

S2 > S3 (5.6)

In each line we might get several NBP candidates, so the final NBP will be the candidate

that has the brightest S2 sector. In order to avoid picking some bright video noise as the NBP

[5] we will be using the horizontal intensity profile; so instead of applying the SSR [1] filter to

a line of the ROI we will be applying it to the horizontal profile calculated from the first line to

the line that we are dealing with, because as already mentioned the values will accumulate

faster at the nose bridge location, so by using the horizontal profile we are sure that we are

picking the right NBP

Candidate not some bright point caused by noise; of course the results will get more

accurate as we reach the last line of the ROI because the accumulation at the nose bridge

location will get more obvious.

Fig 5.6 Nose bridge detection

22


23/70

ROI is enlarged only for a clearer vision. Now that we located the nose bridge we need

to find the nose tip on that bridge. Since each NBP represents the brightest S2 sector on the line

it belongs to, and that S2 sector contains the accumulated vertical sum of the intensities in that

sector from the first line to the line it belongs to, we will be using this information to locate the

nose tip. Nose trills are dark areas, and the portion that they add to the accumulated sum in the

horizontal profile is smaller than the contribution of other areas; in other words each NBP will

add with its S2 sector a certain amount to the accumulated sum in the horizontal profile, but the

NBP at the nose trills location will add a smaller amount (S2 sector of the NBP at the nose trills

location is darker than S2 sector of other NBPs), so if we calculate the first derivate of the

values of NBPs S2 sectors (the first derivate of the values of the maximum value of the

horizontal profile at each ROI line) we will notice a local minima at the nose trills location (see

fig. 5.6); by locating this local minima we take the NBP that corresponds to it as the nose trillslocation, and the next step is to look for the nose tip above the nose trills.

Since the nose tip is brighter than other features it will donate with its S2 sector to the

accumulated sum more than other NBPs [5], which means a local maxima in the first derivate

(see fig. 5.6); so the location of the nose tip is the location of the NBP that corresponds to the

local maxima that is above the local minima in the first derivate.

5.5 Module 3

5.5.1 Face Tracking Module

The nose tip is tracked to use its movement and coordinates as the movement and coordinates

of the mouse pointer. The eyes are tracked to detect their blinks, where the blink becomes the

mouse click.

The tracking process is based on predicting the place of the feature in the current

frame based on its location in previous ones; template matching and some heuristics are applied

to locate the features new coordinates.

5.6 Module 4

5.6.1 Front End

23


24/70

This light weight eye controlled computer has a front end which is created using SWINGS and

AWT. Java swing has many built in APIs for creating the front end. The front end of the eye

controlled computer provides for a good user interface.

Swing is a GUI toolkit for Java. It is one part of the Java Foundation Classes (JFC).Swing includes graphical user interface (GUI) widgets such as text boxes, buttons, split-panes,

and tables. Swing widgets provide more sophisticated GUI components than the earlier

Abstract Window Toolkit. Since they are written in pure Java, they run the same on all

platforms, unlike the AWT which is tied to the underlying platform's windowing system. Swing

supports pluggable look and feel not by using the native platform's facilities, but by roughly

emulating them. This means you can get any supported look and feel on any platform. The

disadvantage of lightweight components is slower execution. The advantage is uniform

behavior on all platforms.

Advantages of Swings:-

Swing is a platform independent, Model-View-ControllerGUI framework for Java. It follows a

single-threaded programming model, and possesses the following traits:

Platform independence: Swing is platform independent both in terms of its expression

(Java) and its implementation (non-native universal rendering of widgets).

Extensibility: Swing is a highly partitioned architecture which allows for the 'plugging'

of various custom implementations of specified framework interfaces: Users can

provide their own custom implementation(s) of these components to override the

default implementations. In general, Swing users can extend the framework by:

extending existing (framework) classes; providing alternative implementations of core

components.

Component-Oriented: Swing is a component-based framework. The distinction

between objects and components is a fairly subtle point: concisely, a component is a

well-behaved object with a known/specified characteristic pattern of behavior. Swing

objects asynchronously fire events, have 'bound' properties, and respond to a well

known set of commands (specific to the component.) Specifically, Swing components

are Java Beans components, compliant with the Java Beans Component Architecture

specifications.

24


25/70

Customizable: Given the programmatic rendering model of the Swing framework, fine

control over the details of rendering of a component is possible in Swing. As a general

pattern, the visual representation of a Swing component is a composition of a standard

set of elements, such as a 'border', 'inset', decorations, etc. Typically, users will

programmatically customize a standard Swing component (such as a JTable) by

assigning specific Borders, Colors, Backgrounds, etc., as the properties of the

component. The core component will then use these properties (settings) to determine

the appropriate renders to use in painting its various aspects. However, it is also

completely possible to create unique GUI controls with highly customized visual

representation. (Swing components support transparent rendering.)

Configurable: Swing's heavy reliance on runtime mechanisms and indirect

composition patterns allow it to respond at runtime to fundamental changes in its

settings. For example, a Swing-based application can change its look and feel at

runtime (from say MacOS look and feel to a Windows XP look and feel). Further, users

can provide their own look and feel implementation, which allows for uniform changes

in the look and feel of existing Swing applications without any programmatic change to

the application code.

Lightweight UI: The magic of Swing's configurability is also due to the fact that it does

NOT use the native host OS's GUI controls for representation, but rather 'paints' its

controls programmatically, through the use of Java 2D api's. Thus, a Swing component

does NOT have a corresponding native OS GUI 'peer', and is free to render itself in any

way that is possible with the graphics APIs.

AWT, Abstract Windowing Toolkit, is a Javas original way of working with graphics.

AWT components are now called heavyweight components because of their significant use of

system resources. AWT is now supplemented with the Swing Package. AWT provides the

foundation of graphics work in Java, and even the Swing Package is based on AWT.

Text Fields

Text fields are the basic text handling components of the AWT. These components handle a

one dimensional string of text; they let you display text, let the user enter text, allow you to

25


26/70

take passwords by masking typed text, allow you to read the text the user has entered, and most

fundamental AWT components.

Buttons

Buttons provide users with a quick way to start some action-all they have to do is click them.

Every user is familiar with buttons, and we have already taken a look behind the scenes on how

buttons work in code when discussing event handling. You can give buttons a caption, such as

Click Me! when the user does click the button. Your code if you have registered to handle

events from the button.

Checkboxes

Checkboxes are much like, buttons, except that they are dual state, which means they canappear as selected or unselected. when selected ,they display a visual indication of some kind

,such as a checkmark or an x(it varies by operating system in AWT programming, which is

one of the reason sun introduced swing, which can display components with the same look

across many operating systems), the user can check a check box to select an option of some

kind (such as the items on sandwich),to enable automatic spell checking, or to enable

automatic spell checking, or to enable printing while he is doing something else. You use

checkbox to let the user select nonexclusive options; for example, both automatic spell

checking and background printing may be enabled at the same time.

Radio buttons

You let the user select one of a set of mutually exclusive options using options using radio

buttons. Only one of a set of option buttons can let the user select a printing color or the day of

the week. In AWT radio buttons are actually a type of checkbox, and when selected, they

display a round dot or a clicked square or some other indication (again, the visual indication

depends on the operating system).you use radio buttons in groups.

Layouts

We have just added components to applets and applications using the add method is actually a

method of the default layout manager-the flow layout manager is res possible for arranges

components in AWT applets, by default. The flow layout manager arranges much like a word

processor might arrange words-across the page and then wrapped to the next line as needed

,creating what sun calls a flow of components. you will see that you can customize flow

layouts to some extend ,such as left center, or right-aligning components .however ,the

26


27/70

limitation of flow layout are clear, especially if you are counting on components maintaining

some position with respect to others, because if user resizes your applet or application ,the

components will all move around. On the other hand, there are other AWT layout manager

(and quite a few new ones in swing), and we all cover the AWT grid, border, card, and grid bag

layout managers.

Why cant you just set where a components goes and forget it? Novice java

programmers are often frustrated by having to deal with AWT layout mangers and want to

know when they cant just give the coordinates for the components they wants to use and be

done with it. In fact, you can, although youre responsible for handling the case where

windows are resized and components should be moved to position components where you want

them, you can indicate that you want no layout manager at all. Then you can size and locatecomponents as you want them, using adds to display them, like this:

Setlayout(null);

text1 = new textfield(20);

text1.setsize(200,50);

text1.setlocaton(20,20);

add(text1);

This adds a text field of size (200, 50) at location (20, 20) in a container such as an

applet window. Therefore, as you can add components to containers without any layout

manager at all, something thats useful to bear in mind if the AWT layouts frustrate you too

much.

One very useful AWT container to use with layouts is the panel components you can

arrange components in a panel and then add the panel, itself, to the layout of an applet or

application.

AWT vs. SWING

Theres a fundamental difference philosophical difference between AWT and swing

components. Each AWT components gets its own operating platform window (and therefore

ends up looking like a standard controlling that operating platform). In extended programs, a

large number of such windows shows performance and uses up a great deal of memory .such

components have come to be heavyweight. Swings controls, on the other hand, are simply

27


28/70

drown as images in their containers and dont have an operating platform window of their own

at all, so they use far fewer system resources .therefore, theyre called lightweight components.

All swings components are derived from the Jcomponents class, and this class is in

turn, derived from the AWT container class, which has no heavyweight itself, so Jcomponent is

a lightweight class. Jcomponent adds a tremendous amount of programming support to the

AWT components class. Note that because all swings components are derived from

Jcomponent, and Jcomponent are derived from the AWT container class, all swing components

are also AWT components in that way, and you can mix AWT controls with swing controls in

your programs.

Not all swing components are lightweight; to display anything windowed environment,

you need some operating system windows, if only to draw lightweight controls in. For thatreason, swing supports these heavyweight classes; JFrame, JDialog, Jwindow, and JApplet.

Applet class and AWT application using the Frame class, you build swing applets on

the JApplet class and swing applications using the JFrame.

28


29/70

Chapter 6

SOURCE CODE

Application.Java

import javax.swing.UIManager;

import java.awt.*;

import c.WaitFrame;

public class Application

{

boolean packFrame = false;

//Construct the application

public Application()

{

WaitFrame wFrame = new WaitFrame();

Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize();

wFrame.setLocation((screenSize.width - wFrame.getWidth()) / 8, (screenSize.height -

wFrame.getHeight()) / 8);

wFrame.setUndecorated(true);

wFrame.getContentPane().setCursor(Cursor.getPredefinedCursor(Cursor.WAIT_CURSOR));

29


30/70

wFrame.show();

Frame frame = new Frame(wFrame);

//Validate frames that have preset sizes

//Pack frames that have useful preferred size info, e.g. from their layout

if (packFrame)

{

frame.pack();

}

else

{

frame.validate();}

//Center the window

Dimension frameSize = frame.getSize();

if (frameSize.height < screenSize.height)

{

frameSize.height = screenSize.height;

}

if (frameSize.width < screenSize.width)

{

frameSize.width = screenSize.width;

}

frame.setLocation((screenSize.width - frameSize.width)/2, (screenSize.height -

frameSize.height)/2 );

frame.setVisible(true);

}

//Main method

public static void main(String[] args)

{

try

{

UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());

}

30


31/70

catch(Exception e) {

e.printStackTrace();

}

new Application();

}

}

ConnectedComponents.java

import java.util.*;

import java.awt.*;

public class ConnectedComponents

{

private int labels[],cluster;

private int label;

private int clustersMembers[][];

public int counters[],indecies[],clusters[];

public ConnectedComponents(int labels[],int clustersMembers[][])

{

label = 0;

this.labels = labels;

this.clustersMembers = clustersMembers;

}

public int findClustersLabels(int x,int y,int width)

{

//The algorithm is the following :

//If the pixel has no neighbors( i.e in a binary image: all neighbors are 0s ) assign a new label

to it

//If the pixel has one neighbor take it's label

31


32/70

//If the pixel has more than one neighbor take one of the labels and mark the labels as similar

int w,nw,n,ne;

if(y == 0)

{

if(x == 0)

{

label++;

clustersMembers[x][y] = label;

labels[label] = Integer.MIN_VALUE;

}

else

{if(clustersMembers[x-1][y] == 0)

{

label++;



}

else

clustersMembers[x][y] = clustersMembers[x-1][y];

}

}

else

{

n = clustersMembers[x][y-1] ;

if( x+1 == width )

ne = 0;

else

ne = clustersMembers[x+1][y-1];

if(x != 0)

{

nw = clustersMembers[x-1][y-1];

w = clustersMembers[x-1][y];

if( (w == 0) && (nw == 0) && (n==0) && (ne==0) )

{

32


33/70

label++;



}

else

{

if (w != 0)

{

clustersMembers[x][y] = w;

if(nw != 0)

{

if( nw != w )labels[nw] = w;

if( (n==0) && (ne!=0) && (nw != ne) && (labels[nw] != ne) )

labels[ne] = nw;

}

else

{

if(n != 0)

{

if( w != n)

labels[n] = w;

}

else

if( (ne != 0) && (ne != w) && (labels[w] != ne) )

labels[ne] = w;

}

}

else

{

if (nw != 0)

{

clustersMembers[x][y] = nw;

if( (n==0) && (ne!=0) && (ne !=nw) && (labels[nw] != ne) )

labels[ne] = nw;

33


34/70

}

else

if (n != 0)

clustersMembers[x][y] = n;

else

if (ne != 0)

clustersMembers[x][y] = ne;

}

}

}

else

{if( (n==0) && (ne==0) )

{

label++;



}

else

{

if( n != 0)

clustersMembers[x][y] = n;

else

clustersMembers[x][y] = ne;

}

}

}

labels[label+1] = Integer.MAX_VALUE; //a flag that shows where the array ends

return label;

}

////////////////////////////////////////

public void findRootLabels() //For equivalent labels, find the representitive label

{

34


35/70

clusters = new int[label + 2];

cluster = 0;

if( label == 1 ) //one label found

{

cluster++;

clusters[cluster] = 1;

clusters[cluster + 1] = Integer.MAX_VALUE;

}

else

{

boolean visited[] = new boolean[label+1];

int j,i;//eliminate loops and find roots

for(i=1; labels[i] != Integer.MAX_VALUE ; i++)

{

if( labels[i] != Integer.MIN_VALUE )

{

for (int q = 1; q < visited.length; q++)

visited[q] = false;

visited[i] = true;

j = i;

while (labels[j] != Integer.MIN_VALUE)

{

j = labels[j];

if (visited[j])

{

labels[i] = Integer.MIN_VALUE;

break;

}

else

visited[j] = true;

}

labels[i] = j;

}

else

35


36/70

{

cluster++;

clusters[cluster] = i;

}

}

for (i = 1; labels[i] != Integer.MAX_VALUE ; i++)

if (labels[i] == Integer.MIN_VALUE)

labels[i] = i;

clusters[cluster + 1] = Integer.MAX_VALUE;

}

}

/////////////////////////////////////////

private int[][] findClustersCenters(int clusters[],double limit,int width,int height)

{

if( cluster == 1 ) //one cluster found

{

int centerX = 0;

int centerY = 0;

int counter = 0;

for (int y = 0; y < height; y++)

{

for (int x = 0; x < width; x++)

{

if (clustersMembers[x][y] != 0)

{

centerX += x;

centerY += y;

counter++;

}

}

}

if( counter > limit ) //if cluster size is larger than the threshold

{

36


37/70

int centers[][] = new int[cluster+2][2];

centers[1][0] = (int)centerX / counter;

centers[1][1] = (int)centerY / counter;

centers[cluster+1][0] = Integer.MAX_VALUE;

return centers;

}

else

return null;

}

else

{

//more than one clusterlong[] centerX = new long[cluster + 2];

long[] centerY = new long[cluster + 2];

counters = new int[cluster + 2];

indecies = new int[label + 2];

int index, clusterArea;

for (int i = 1; clusters[i] != Integer.MAX_VALUE; i++)

indecies[clusters[i]] = i;

Vector finalClusters = new Vector();


{


{


{

index = indecies[labels[clustersMembers[x][y]]];

centerX[index] += x;

centerY[index] += y;

counters[index]++;

}

}

}


37


38/70

{

clusterArea = counters[indecies[clusters[i]]];

if (clusterArea > limit)

finalClusters.add(new Integer(clusters[i]));

else

cluster--;

}

if( cluster != 0 )

{

for (int i = 0; i < finalClusters.size(); i++)

clusters[i + 1] = ( (Integer) (finalClusters.get(i))).intValue();

clusters[cluster + 1] = Integer.MAX_VALUE;int centers[][] = new int[cluster + 2][2];


{

index = indecies[clusters[i]];

centers[i][0] = (int) (centerX[index] / counters[index]);

centers[i][1] = (int) (centerY[index] / counters[index]);

}

centers[cluster + 1][0] = Integer.MAX_VALUE;

return centers;

}

else

return null;

}

}

//////////////////////////////////////////////////

private Point oneClusterCenter(int centerX,int centerY,int counter,int start

,int end,int width,double limit)

{

for (int y = start; y < end; y++) //look in the specified region

{

38


39/70


{


{

centerX += x;

centerY += y;

counter++;

}

}

}

if (counter > (double) limit / 2d)

{Point center = new Point();

center.setLocation( (int) (centerX / counter), (int) (centerY / counter));

return center;

}

else

return null;

}

//////////////////////////////////////////////////

private Point multipleClustersCenter(int clusters[],int start,int end,int width,double limit

,int grayPixels[],int x0,int y0,int fWidth)

{

long[] centerX = new long[cluster + 2];

long[] centerY = new long[cluster + 2];

long[] colors = new long[cluster + 2];

int[] counter = new int[cluster + 2];

int[] indecies = new int[label + 2];

int index = 0, clusterArea, darkestPixel = 256, pixelColor, darkestX = 0, darkestY = 0, ind;


indecies[clusters[i]] = i;

for (int y = start; y < end; y++)

{

39


40/70


{


{

index = indecies[labels[clustersMembers[x][y]]];

centerX[index] += x;

centerY[index] += y;

counter[index]++;

pixelColor = grayPixels[ (y + y0) * fWidth + (x + x0)];

colors[index] += pixelColor;

if (pixelColor < darkestPixel)

{darkestPixel = pixelColor;

darkestX = x;

darkestY = y;

}

}

}

}

double max = Double.NEGATIVE_INFINITY, ratio, dist = 0;

int cX, cY;

index = Integer.MIN_VALUE;


{

clusterArea = counter[indecies[clusters[i]]];

if (clusterArea > (double) limit / 2d)

{

ind = indecies[clusters[i]];

cX = (int) (centerX[ind] / counter[ind]);

cY = (int) (centerY[ind] / counter[ind]);

dist = Math.sqrt(Math.pow( (cX - darkestX), 2) +

Math.pow( (cY - darkestY), 2));

ratio = clusterArea / (colors[ind] * dist); //look for the largest,darkest,and closest to the

darkest pixel, cluster

if (ratio > max)

40


41/70

{

max = ratio;

index = ind;

}

}

}

if (index != Integer.MIN_VALUE)

{

Point center = new Point();

center.setLocation( (int) (centerX[index] / counter[index]),

(int) (centerY[index] / counter[index]));

return center;}

else

return null;

}

//////////////////////////////////////////////////

public Point findPupilsClustersCenters(int x0,int y0,int clusters[],double limit,int width,int

height,int grayPixels[],int fWidth)

{

if( cluster == 1 ) //one cluster found

{

int centerX = 0;

int centerY = 0;

int counter = 0; //look in the lower half

Point center = oneClusterCenter(centerX,centerY,counter,(int)height/2,height,width,limit);

if( center != null )

return center;

else

{

centerX = 0;

centerY = 0;

counter = 0; //look in the upper half

41


42/70

return oneClusterCenter(centerX,centerY,counter,0,(int)height/2,width,limit);

}

}

else

{ //look in the lower half

Point center = multipleClustersCenter(clusters,

(int)height/2,height,width,limit,grayPixels,x0,y0,fWidth);

if( center != null )

return center;

else //look in the upper half

return multipleClustersCenter(clusters,0,

(int)height/2,width,limit,grayPixels,x0,y0,fWidth);}

}

//////////////////////////////////////////////////

public int[][] processClusters(double limit,int width,int height)

{

findRootLabels();

return findClustersCenters(clusters,limit,width,height);

}

public int getCluster() {

return cluster;

}

public int getLabel() {

return label;

}

///////////////////////////////////////////////////

}

42


43/70

FaceDetector.java

import java.util.*;

import java.awt.*;

import svm.*;

public class FaceDetector

{

final int fWidth = 320, fHeight = 240;

int[] labels, start, binaryPixels;

public int[] grayPixels,pixels;

int[][] s, ii, clustersMembers, eyes;

int faces, cluster;

svm_model model;

svm_node nodes[];

//////////////////////////////

public FaceDetector(svm_model model)

{

pixels = new int[fWidth * fHeight];

grayPixels = new int[fWidth * fHeight];

binaryPixels = new int[fWidth * fHeight];

s = new int[fWidth][fHeight];

ii = new int[fWidth][fHeight];

clustersMembers = new int[fWidth][fHeight];

labels = new int[fWidth * fHeight];

nodes = new svm_node[735];

this.model = model;

start = new int[model.nr_class];

start[0] = 0;

for (int i = 1; i < model.nr_class; i++)

start[i] = start[i - 1] + model.nSV[i - 1];

43


44/70

}

////////////////////////////////////////

public int[] detectAllFaces(Image image)

{

//Initializing

pixels = ImageProcessing.extractPixels(image,0,0,fWidth, fHeight, pixels);

grayPixels = ImageProcessing.toGrayscale(pixels, grayPixels);

binaryPixels = ImageProcessing.toBinary(grayPixels,binaryPixels,128);

ii = ImageProcessing.calculateIntegralImage(fWidth, fHeight, grayPixels, s, ii);

faces = 0;int coords[];

//Apply biggest filter first

// SSRFilter filter1 = new SSRFilter(120,72, ii);

// coords = detectFaces(filter1);

//If no faces were found with the previous filter apply a smaller one, and so on...

// if (faces == 0)

{

SSRFilter filter2 = new SSRFilter(84, 54, ii);

coords = detectFaces(filter2);

}

if (faces == 0)

{

SSRFilter filter3 = new SSRFilter(60, 36, ii);

coords = detectFaces(filter3);

}

return coords;

}

////////////////////////////////////////

private int[] detectFaces(SSRFilter filter)

44


45/70

{

//Detect faces with the given filter

//If the pixel is a face candidate find it's cluster

int xCor, yCor, label = 0;

for (int i = 0; i < fWidth; i++)

for (int j = 0; j < fHeight; j++)

clustersMembers[i][j] = 0;

ConnectedComponents CC = new ConnectedComponents(labels, clustersMembers);

for (int y = 0; y < fHeight - filter.getHeight(); y++)

for (int x = 0; x < fWidth - filter.getWidth(); x++){

xCor = x + 1 + (filter.getWidth() / 2); //+1 : transition error in sectors calculation

yCor = y + 1 + (filter.getHeight() / 2);

if (filter.foundFaceCandidate(x, y) &&

ImageProcessing.isSkinPixel(pixels[yCor * fWidth + xCor]))

label = CC.findClustersLabels(xCor, yCor,fWidth);

}

//If there are face candidates

if (label != 0)

{

//Find root labels,and their centers

int centers[][] = CC.processClusters(filter.getArea()/48d,fWidth,fHeight);

//If there are valid clusters

if (centers != null)

{

//Find left/right pupils for each of the candidates

cluster = CC.getCluster();

int counter = findPupilsCandidates(filter,centers);

//if found some eyes

if( counter != -1 )

45


46/70

{

//Load templates to the array in order to classify them

Vector templates = loadEyesTemplates(counter);

//Classiffying templates

double cResults[] = classify(templates);

//Multiply classification results by clusters' areas

if( cluster > 1 )

cResults = multiplyByArea(CC.indecies,CC.clusters,CC.counters,cResults);

//Find the template with the highest scoreint face[] = findBestEyesTemplate(cResults);

//Find nose tip

if( face != null )

{

Point noseTip = findNoseTip(face);

if( noseTip != null )

{

int coordinates[] = new int[6];

coordinates[0] = face[0];




coordinates[4] = (int)noseTip.getX();

coordinates[5] = (int)noseTip.getY();

return coordinates;

}

}

}

}

}

return null;

}

46


47/70

//////////////////////////////////////////////

private Point findPupil(int x0, int y0,int width,int height,int labels[], int pupilsMembers[]

[],int limit)

{

int label = 0; //binarize the sector and look for the appropriate cluster



pupilsMembers[x][y] = 0;

ConnectedComponents CC = new ConnectedComponents(labels,pupilsMembers);

for (int y = 0; y < height; y++)for (int x = 0; x < width; x++)

if( binaryPixels[ (y + y0) * fWidth + (x + x0)] == 1 )

label = CC.findClustersLabels(x, y, width);

if( label != 0 )

{

CC.findRootLabels();

return

CC.findPupilsClustersCenters(x0,y0,CC.clusters,limit,width,height,grayPixels,fWidth);

}

else

return null;

}

/////////////////////////////////////////////

private int findPupilsCandidates(SSRFilter filter,int centers[][])

{

eyes = new int[cluster][5];

int width = filter.getWidth()/3;

int height = filter.getHeight()/2;

int labels[] = new int[width*height];

int pupilsMembers[][] = new int[width][height];

int limit = (int)filter.getArea()/(144*6);

47


48/70

int halfW, halfH, sixthW, counter = -1;

halfW = filter.getWidth() / 2;

halfH = filter.getHeight() / 2;

sixthW = filter.getWidth() / 6;

Point lp1,rp1;

int x, y;

for (int i = 1; centers[i][0] != Integer.MAX_VALUE; i++)//pass all clusters

{

x = centers[i][0];

y = centers[i][1];

//find left pupil in sector s1

lp1 = findPupil(x-halfW,y-halfH,width,height,labels,pupilsMembers,limit);if( lp1 == null )

continue;

//find right pupil in sector s3

rp1 = findPupil(x+sixthW,y-halfH,width,height,labels,pupilsMembers,limit);

if( rp1 == null )

continue;

counter++;

eyes[counter][0] = (int)lp1.getX()+x-halfW;

eyes[counter][1] = (int)lp1.getY()+y-halfH;

eyes[counter][2] = (int)rp1.getX()+x+sixthW;

eyes[counter][3] = (int)rp1.getY()+y-halfH;

eyes[counter][4] = i;

}

return counter;

}

//////////////////////////////////////////

private svm_node[] extractEyesTemplate(int x0,int y0,int x1,int y1)

{

int xLen,yLen,sX,sY,cX,cY,oX,oY;

48


49/70

double xInc,yInc,nY,scale;

svm_node node;

svm_node template[] = new svm_node[735];

xLen = x1-x0;

yLen = y1-y0;

xInc = (double)xLen / 23d;

yInc = (double)yLen / 23d;

scale = Math.sqrt(Math.pow(xLen,2)+Math.pow(yLen,2))/23d;

oX = x0 - (int)(6*xInc) + (int)(8*yInc);

nY = (yLen != 0 ? (8*yInc*xLen) / yLen : 8*scale);

oY = y0-(int)nY;

sX = oX;sY = oY; //derotate the template to a horizontal position

for (int y = 0; y < 21; y++)

{

cX = sX;

cY = sY;

for (int x = 0; x < 35; x++)

{

node = new svm_node();

node.index = y * 35 + x + 1;

if ( (cX >= 0) && (cX < fWidth) && (cY < fHeight) && (cY >= 0) )

node.value = grayPixels[cY * fWidth + cX] / 255d;

else

if( y == 0 )

node.value = 128d/255d;

else

node.value = template[ (y - 1) * 35 + x].value;

template[y * 35 + x] = node;

cX = sX+(int)((x+1)*xInc);

cY = sY+(int)((x+1)*yInc);

}

sX = oX-(int)((y+1)*yInc);

sY = oY+(int)((y+1)*xInc);

}

49


50/70

return template;

}

//////////////////////////////////////////

private Vector loadEyesTemplates(int counter)

{

svm_node[] template;

Vector templates = new Vector();

//Extract the templates and write them in the 'LibSVM' file format

for (int i = 0; i


51/70

//////////////////////////////////////////////////

private double[] multiplyByArea(int indecies[],int clusters[],int counters[],double cResults[])

{

for (int i = 0; i < cResults.length; i++)

if( cResults[i] > 0 )

cResults[i] *= counters[indecies[clusters[eyes[i][4]]]];

return cResults;

}

//////////////////////////////////////////////////

private int[] findBestEyesTemplate(double cResults[])

{

double max = 0d;

int index = 0;

for (int i=0 ; i< cResults.length ; i++)

if( cResults[i] > max )

{

max = cResults[i];

index = i;

}

if( max>0 )

{

faces++;

int face[] = new int[4];

face[0] = eyes[index][0];




return face;

}

else

51


52/70

return null;

}

//////////////////////////////////////////////////

private Point[] findNoseBridge(int length,int ii[][])

{

int val,max,nbcx = 0,counter = 0;

SSRFilter filter = new SSRFilter(length/2,1,ii);

Point candidates[] = new Point[length];

for(int y=0 ; y < length ; y++){

max = Integer.MIN_VALUE;

for(int x=0 ; x < length-filter.getWidth() ; x++)

{

val = filter.findNoseBridgeCandidate(x,y);

if( val > max )

{

max = val;

nbcx = x + 1 + (filter.getWidth() / 2);

}

}

if( max != Integer.MIN_VALUE )

{

counter++;

candidates[y] = new Point(nbcx, max);

}

}

if( counter > (length/5) )

return candidates;

else

return null;

}

52


53/70

////////////////////////////////////////////////

private int findNextCandidate(int i,Point candidates[])

{

while( (candidates[i] == null) && (i < candidates.length-1) )

{

i++;

}

return i;

}

////////////////////////////////////////////////

private int[] calculateGradiants(Point candidates[],int length)

{

int gradiants[] = new int[length],i=0,ind1=0,ind2=0;

for (int q = 0; q < gradiants.length; q++)

gradiants[q] = Integer.MAX_VALUE;

Point c1,c2;

while( true )

{

ind1 = findNextCandidate(ind2,candidates);

c1 = candidates[ind1];

if( c1 != null )

{

if (ind1 + 1 < length)

{

ind2 = findNextCandidate(ind1 + 1, candidates);

c2 = candidates[ind2];

if (c2 != null)

{

gradiants[ind2] = (int)(c2.getY() - c1.getY());

if( ind2 == length-1 )

break;

}

53


54/70

else

break;

}

else

break;

}

else

break;

}

return gradiants;

}

//////////////////////////////////////////////////

private Point findNoseTip(int face[])

{

int x1,y1,x2,y2,xLen,yLen,length,step,sX,sY,cX,cY;

double slope;

x1 = face[0];

y1 = face[1];

x2 = face[2];

y2 = face[3];

xLen = x2-x1;

yLen = y2-y1;

length = (int)Math.sqrt(Math.pow(xLen,2)+Math.pow(yLen,2));

step = Math.abs(yLen != 0 ? xLen/yLen : xLen);

step = ( step < 3 ? 3 : step );

slope = ( yLen < 0 ? -1 : 1);

int ROI[] = new int[length*length];

sX = x1;

sY = y1;

for (int y = 0; y < length; y++) //extract face 'Rigion Of Interest'

{

cX = sX;

cY = sY;

for (int x = 0; x < length; x++)

54


55/70

{

if ( (cX >= 0) && (cX < fWidth) && (cY < fHeight))

ROI[y * length + x] = grayPixels[cY * fWidth + cX];

else

ROI[y * length + x] = ROI[(y-1)*length+x];

cX++;

if ( (x + 1) % step == 0)

cY = (int) (cY + slope);

}

if ( (y + 1) % step == 0)

sX = (int) (sX - slope);

sY++;}

int ii[][] = new int[length][length];

int s[][] = new int[length][length];

ii = ImageProcessing.calculateIntegralImage(length,length,ROI,s,ii);

Point candidates[] = findNoseBridge(length,ii); //find nose bridge candidates

if( candidates != null )

{

int gradiants[] = calculateGradiants(candidates, length);

int uMin1 = Integer.MAX_VALUE,uMin2 = Integer.MAX_VALUE,

lMin1 = Integer.MAX_VALUE,lMin2 = Integer.MAX_VALUE;

int lMinGrad,lMinIndex,uMinGrad,uMinIndex,minGrad,minIndex;

int uInd1=0,uInd2=0,lInd1=0,lInd2=0,gLength;

gLength = gradiants.length;

for(int i=0 ; i


56/70

uInd2 = i;

}

for(int i=gLength/2 ; i= 0.5 )

{

uMinGrad = uMin2;

uMinIndex = uInd2;

}

else

{

uMinGrad = uMin1;

uMinIndex = uInd1;

}

if( (double)uMinGrad/(double)lMinGrad >= 0.5 )

56


57/70

{

minGrad = lMinGrad;

minIndex = lMinIndex;

}

else

{

minGrad = uMinGrad;

minIndex = uMinIndex;

}

int start;

if( minIndex >= gLength/2)

{if (minIndex < 3 * gLength / 4)

start = gLength / 2;

else

start = 3 * gLength / 4;

}

else

start = 0;

int max = 0,index = 0;

for( int i=start ; i max) && (gradiants[i] != Integer.MAX_VALUE) ) //gradiant

above it ( nose tip )

{

max = gradiants[i];

index = i;

}

if( candidates[index] == null )

return null;

Point noseTip = new Point((int)candidates[index].getX(),index);

slope = (double)yLen / (double)xLen;

double angle = Math.atan(slope); //rotate ROI to it's original state

double x = Math.cos(angle)*noseTip.getX()-Math.sin(angle)*noseTip.getY();

double y = Math.sin(angle)*noseTip.getX()+Math.cos(angle)*noseTip.getY();

57


58/70

x += face[0];

y += face[1];

noseTip.setLocation(x,y);

return noseTip;

}

return null;

}

Chapter 7

TESTING

7.1 Introduction to testing

Testing is a process used to help identify the correctness, completeness and quality of

developed computer software. With that in mind, testing can never completely establish the

correctness of computer software it just find out the faults whatever we did in our project.

Testing helps is verifying and Validating if the Software is working as it is intended

to be working. This involves using Static and Dynamic methodologies to Test the application.

Software

spends more time being maintained than being developed.

Things that

were obvious in development are easily obscured.

Fixing bugs iseasier than finding them and making sure nothing else breaks.

The real cost

of software is programmer time and effort, not length of code, licenses or required

hardware.

7.2 Test Cases

58


59/70

Test cases are a well define Scenario to perform a task to validate the Inputs and outputs. Test

cases are a sequence of steps to test the correct behavior of a functionality/feature of an

application.

The test cases are given below.

Table 6.2 Sample Test Cases to check the Functionality of the Application

59


60/70

Scenario ID Test case

Description

Steps Expected

Results

Actual

Results

LIVE VIDEO

CAPTURE

Test whether the

live video is

captured in the

captured image

segment.

1.Installing the

webcam

2.Initializing the

JMF

1.Live video

is detected

2. All the

connected

devices will

be detected.

1. Live video

is detected

2. All the

connected

devices will

be detected

FACE

DETECTION

Test whether the

face is detected

1. Click the

DETECT FACE

button.

1. Face is

detected

2.Detecting

the facial

features

1. Face is

detected

2.Detecting

the facial

features(nose

Eyebrows,

Eyes)

FACE

TRACKING

To track the face

movement while

changing the

location of face

1. Movement of

capture figure is

necessary

1. The facial

features are

to be

detected

while

changing the

location of

the face

1The facial

features are

not exactly

detected

while

changing the

location of

the face

ENABLE THE

INTERFACE

To test 1. Click

ENABLE

INTERFACE

1. While click

the button, the

mouse is

replaced by our

facial features

1. While click

the button, the

mouse is

replaced by our

facial features

60


61/70

Chapter 8

CONCLUSION AND FUTURE ENHANCEMENTS

8.1 Conclusion

We have successfully implemented the project. Thus we have developed a project capable of

replacing mouse with facial features to interact with computer.

The main advantage of this project is that only a simple webcam with moderate

resolution is sufficient to capture the live video. People with hand disabilities can use this

project effectively.

The disadvantage of this project is that involuntary eye blinks also trigger mouse clicks.

Also it may lead to strain in the eyes due to continuous blink.

8.2 Future Enhancement

Involuntary eye blinks can be detected and they can be prevented from firing the events.

The sensitivity of nose movement is very high that even a small reaction induces

greater mouse movement.

61


62/70

Appendix A

SNAPSHOTS

A.1 Login GUI

62


63/70

A.2 Detecting the Face

63


64/70

A.3 Face Tracking

64


65/70

A.4 Enabled Visage

65


66/70

Appendix B

ACRONYMS AND ABBREVATIONS

66


67/70

BTE Between the eyes

HCI Human Computer Science

JMF Java Media Framework

SSD Squared Difference

SSR Six-Segmented Rectangular Filter

NBP Nose Bridge Point

GUI Graphical User Interface

AWT Abstract Windowing Toolkit

JAR Java Archive File

Appendix c

USER MANNUAL

67


68/70

Installing the Java Standard Edition Development Kit (JDK) 5

Our application is developed on the java standard edition development kit (JDK) 5. You can

download the latest version of JDK5 and its documentation from

Java.sun.com/javase/5/download.jsp

After downloading the JDK installer, double click the installer program to begin

installing the JDK. We recommended that you accept all the default installation options. If you

change the default installation directory, be sure to write down the exact name and location of

the directory you choose, as you will need this information later in the installation process on

Windows JDK is placed in the following directory by default:

C:\Program Files\Java\jdk.5.0

Installing the Java Media Framework 2.1.1e

For the installation of the JMF first we have to download the JMFs, version 2.1.1e, installation

file and after downloading the JMF installer, double click the installer program to begin

installing the JMF. We recommended that you accept all the default installation options.

Installing the web camera drivers

To install the driver, we need the setup file of the web camera drivers. We can download or we

can use the installation CD which contains the setup files. And then, double click the installer

program to begin installing the drivers. We recommended that you accept all the default

installation options.

Running the application

68


69/70

To run the application user need to save the application JAR file in our computers secondary

memory, and then we have to click on the JAR file, then we get login GUI. Through the GUI

the user can operate the application. The GUI, having option,

Detect the Face- this key is used to tracking the face with the help of users eye blink and nosetip.

Enable Visage- This key is used to enable the application for the user who got tracked. After

enabling the visage the user is ready to control the computer.

Stop Tracking- This key is used to stop the tracking. This option is useful when the user want

to replace them self to another user before enabling visage.

Refresh Preview- This button is used to refresh the preview.

BIBLIOGRAPHY

69


70/70

REFERENCES

[1] E.Hjelmas and B.K.Low(2001), Face Detection: A survey, Computer Vision and Image

Understanding.

[2] S.Kawato and N. Tetsutani( January 2002), Real-time detection of Between-the-Eyes with

a Circle Frequency Filter: The 5th Asian Conference on Computer Vision,Melbourne,

Australia.

[3] S.Kawato and J.Ohya,(October 2000) Two-step Approach for Real- Time Eye Tracking

with a New Filtering Technique: IEEE Int. Conf. on Systems, Man & Cybernetics, Nashville,

Tennessee, USA.

[4] P.Viola and M.Jones(2001), Rapid object Detection using a Boosted Cascade of Simple

Features, Proc. of IEEE Conf.CVRP.

[5] J.Yang and A.Waibel(1996), A real-time face tracker, Proc.3rd IEEE Workshop on

Application of Computer Vision.

[6] AT&T Laboratories Cambridge, The ORL face

database,http://www.uk.research.att.com/facedatabase.html

[7] http://s.i-techonline.com/Book/Humanoid-Robots-New-Developments/ISBN978-3-902613-

00-4hrnd14.pdf

URLs:

WWW.WIKIPEDIA.ORG

WWW.SOURCEFORGE.NET

WWW.HOWSTUFFWORKS.COM
http://www.us.design-reuse.com/exit?url=http://www.uk.research.att.com/facedatabase.htmlhttp://www.us.design-reuse.com/exit?url=http://www.uk.research.att.com/facedatabase.htmlhttp://www.wikipedia.org/http://www.sourceforge.net/http://www.howstuffworks.com/http://www.wikipedia.org/http://www.sourceforge.net/http://www.howstuffworks.com/http://www.sun.java.com/http://www.us.design-reuse.com/exit?url=http://www.uk.research.att.com/facedatabase.html

Documentation_Vision Based Game Development Using Human

Documents

Transcript of Documentation_Vision Based Game Development Using Human