Work completion seminar defence

OPTIMIZATION OF

GESTURE RECOGNITION

IN ITERACTIVE DIGITAL

SPACE

Mahdi Babaei1131600043

Research scholar – TM R&D Project

MASTER OF SCIENCE IN CREATIVE MULTIMEDIA

Supervisor: Assoc. Prof.Dr.Wong Chee Onn

Co-supervisor: Dr.Lim Yan Peng (Forest)

Work Completion Seminar

INTRODUCTION

OVERVIEW OF PRESENTATIONIntroductionResearch ObjectivesResearch QuestionsLiterature reviewMethodologiesSolution proposalResults and AnalysisConclusion

INTRODUCTION With growth in number of innovation in

new devices on 1990th HCI came to daily life.

From simple traditional GUI

To

Inconvenien

t &

unnatural

3D Virtual

SpaceHuman

GestureDigital

content in

curve

displays

RESEARCH QUESTION “How to optimize the best combination of gesture recognition methods in order to have an efficient system (from user view) while improving the quality of interaction and human factors, in a digital space?”

OBJECTIVES To study on gesture recognition methods and

design a taxonomy to categorize them. To study on Digital space, content and

projection and design a space based on the knowledge.

To study on virtual reality based interaction and environment, pick factors and design a suitable interaction based on that.

To test interactive gesture recognition prototype in virtual space with digital contents.

To solve the disadvantages of proposed combination method of gesture recognition in virtual space with digital content.

LITERATURE REVIEW

HUMAN GESTURES AND TYPES Gestures are expressive, meaningful

body motions involving physical movements of the fingers, hands, arms, head, face, or body.

Types based on(Billinghurst and Buxton 2011):

Gestures in everyday World.Gestures only for interfaces.

There are many taxonomies in this area like: (Kammer, Keck et al. 2010) (Karam 2005)

TAXONOMY OR CATEGORY ? We needed to design a new taxonomy of

gestures because:We need to go deep down in order to find

the exactly suitable gesture category that would match the criteria and category can cover basics only.

Taxonomy provides vocabulary and Tree-Node system and flexibility of adding new vocabularies. On the other hand category can offer a group of nodes with strict hierarchy.

QUALITATIVE HUMAN FACTORS OF GESTURE RECOGNITION

Based on (Barclay, Wei et al. 2011) :Fatigue (Barclay, Wei et al. 2011)

Intuitiveness (Nielsen, Störring et al. 2004)

UsabilityLearnability (Valkov, Steinicke et al. 2010)

Easiness to navigate and orientate (Valkov, Steinicke et al. 2010)

Naturalness and memorability (Lenman,

Bretzner et al. 2002).Ergonomic

QUANTITATIVE HUMAN FACTORS OF GESTURE RECOGNITION

Gesture time and duration(speed) (Barclay, Wei et al. 2011

Accuracy, Precision and error rate User Cooperation

GESTURE RECOGNITION DEVICES

Sensors

Mix

Vision Base

d Methods

and cameras

INTERACTION TYPES 2D Interaction

Two-Dimensional interaction in 3D world.

Three-Dimensional interaction

3D INTERACTION REQUIREMENTS 3D Interaction needs 3D space where

user would be able to have movements.

It needs 3D contents (output) to let user manipulate with them.

It needs 3D hardware tools (input) to detect depth value

HUMAN FACTORS OF INTERACTION

based on (Liu and Shrum 2002,Benyon 2010):Control desire (Burger and Cooper 1979, Liu and Shrum 2002)

AccessibilityComputer-mediated communication

apprehension (CMCA) (Liu and Shrum 2002)(The level of expertise in using computers)

Usability: Based on (Standardization 1998, van Kuijk 2012

): Effectiveness Efficiency Satisfaction

Acceptability

INTERACTIVITY OF GESTURES IN DIGITAL SPACES Based on (Hale and Stanney 2002) actions can

be:

Navigating through space.

Specifying item of interest.

Manipulating objects in the environment.

Changing object values.

Controlling virtual objects.

Issuing task-specific commands.

VIRTUAL REALITY ENVIRONMENTS AND DEVICES

Head-Mounted Displays

Cave Virtual Environment

Virtual re

ality workbench

http://electronics.howstuffworks.com/gadgets/other-gadgets/virtual-reality-pictures.htm

HUMAN FACTORS OF VIRTUAL ENVIRONMENTS

Based on (Wilson and D’Cruz 2006) :

Influence of interaction on both sides.

User’s characteristics.

User’s needs.

METHODOLOGY AND PROPOSED SOLUTION

DESIGNED TAXONOMY

Virtual

Physical(epistemic)

Two-degree freedom for 2D interactionMultiple DOF for 2D interactionMultiple DOF for 3D interaction

Gestures with tangible objects for 3D interaction

Gestures for Real-World physical object interaction

Paralinguistic

Linguistic

Act

Symbol

Deictic Mimetic Gesticulations Metaphoric Affect displays Beat

Referential

Modelizing

Descriptive

Suggestive

Prompting

Emphatic

Side Effects of expressive behaviours

Mix-Communication Symbolic-Interactive

Human Gestures

Symbolic (based on One-Way Communication)

(Communicative or Semiotic)

Interactive (based on Two-Way communication) or

Manipulative Communication

Emblems/Illustrators

Iconic

Regulators

GESTURE DESIGN

Electromagneti

cAcoustic Optical Mechanical

Advantage - - Fast upload rate -

Disadvantag

e

High inference with

magnetic field

Low rate target

positioning

Sight can obscured or

interfered

Limits user’s range

of motion

3D Model BasedAppearance-based

Volumetric Skeletal

Speed

Low

(complexity

of calculation)

High

( only key parameters

are analyzed)

Medium

(depends on algorithm)

Accuracy High High Medium

Processing

timeHigh Low Low

Points complicated 3D surfaces Skeleton Joints extraction Shape extraction

SYSTEM DESIGN

Three-Dimensional

Virtual Environment

Gesture Recognition

Interaction

Two-Dimensional interaction in 3D

world

Two-Dimensional

3D- Hardware

3D- Space, 3D- Contents

System Control

Navigation

Selection

Object Manipulation Physical Movement

Manual viewport Manipulation

Steering

Target-based Travel Route planning

FACTOR INTERSECTION FINDING

Accessibility

Computer-mediated communication apprehension

Usability

• Effectiveness• Efficiency• Satisfaction

AcceptabilityInfluence on participantsParticipant’s InfluenceUser Characteristics and needs

Fatigue

Intuitiveness

Usability• Learnability• Easiness to

navigate and orientate

• Naturalness and memorability

• Ergonomic

Speed

Accuracy

User Cooperation

Virtual Environment

Gesture Recognition

Interaction

Control Desire

PROPOSED FRAMEWORK

Gesture recognition factors

Quantitative Factors Qualitative Factors

User expertise in using computer systems

Usability

FatigueIntuitiveness

User cooperation

Speed

Accuracy

Acceptability

Learnability

Easiness to navigate and orientate

Naturalness and memorability

Ergonomic

Control Desire

Accessibility

Satisfaction

Efficiency

Effectiveness

CHOOSING KINECTDevice Popularit

yMotor Driver SDK Image Quality Size Weight Power

Microsoft Kinect

High Has HQ HQ Medium 12"x 3" x 2.5"

3.0 lb Ac + DC

ASUS Xtion /

PrimeSense Carmine

Low

No LQ LQ HQ 7" x 2" x 1.5"

0.5 lb DC-USB

We choose to optimize Microsoft because: It has higher driver quality. It has software development kit.Popularity means easier access to research

resources.

KINECT WEAKNESS Can not track user’s eye and head

movements and rotations

PROPOSED SOLUTION A combination of Microsoft Kinect as

skeletal detection device and an acceleration or gyroscope data.

PROPOSED VE DESIGN

Normal V

iew

Maximize

d View

Cave Design

Implemented C

ave

PROPOSED GESTURE RECOGNITION STRUCTURE

Head and eye

Gestures

BodyGestures Kinect

Camera

AccelerometerDigital Receiver

Analogue Receiver

Analogue to Digital convertor

Antennas

ROTATION AND MOVEMENTS

Yaw

Pitch

Roll

Top/Left/ Bottom/ Right

PROPOSED GESTURE

Knee Heigh

t

Time

PROPOSED HARDWARE

RESULTS AND ANALYSIS

MEASUREMENT STATES Before(Using Microsoft Kinect Only) After(Using proposed combination)

FACTORS Quantitative based on:

Logical optimization.Optimization measurement based on

results.

QualitativeQuestionnaire

QUANTITATIVE MEASUREMENTS Speed

16.6% improvement in coverage angle

Min Max

-100

-60

-20

20

60

100

-60

60

-90

90

Before After

Degre

e

SPEED

Speed0

100

200

300

400

500

527.64

315.32

418.37

AHRS Kinect Proposed method

Rot

atio

n sp

eed

in o

ne

seco

nd(D

egre

e P

er S

econ

d)

Percentage0

102030405060708090

10099.38

59.39

78.80

AHRS Kinect Proposed method

Per

cent

age

of s

ucce

ssfu

l re

cogn

ized

ges

ture

103.05(d/s) improvement in

mean value average of

rotation speed (Degree Per

Second)

19.41 % improvement in mean value of successfully

recognized gestures in a

second

ACCURACY

Error rate percentage in one second0

102030405060708090

100

0.62

40.61

21.20

AHRS KinectProposed Combination

Err

or r

ate

Per

cent

age

19.41 % Reduction in mean value average of error rate.

QUALITATIVE FACTORS Questionnaire:

100 participant. 104 question (each factor 8

question).Same participant completed the

same questionnaire.Two steps: Before and AfterLikert 7 ScaleMean Value test

RESULTS TEST

Reliability StatisticsCronbach's

Alpha N of Items

0.710 104

Reliability StatisticsCronbach's

Alpha N of Items0.752 104

Reliability Test

BEFORE

AFTER

FACTOR 1- EVALUATION OF USER EXPERTISE IN USING COMPUTERS

Mean Average Standard Deviation Average

01234567

4.93

1.76

Overal User Expertiese

Like

rt 7

Sca

le

FACTOR 2 – USER’S FATIGUE

Mean Average Standard Deviation Average0

1

2

3

4

5

6

7

3.18

1.50

2.66

1.12

Before After

Like

rt 7

Sca

le

7.42 % Reduction in mean value average

FACTOR 3 – USER’S ERGONOMIC AND ANXIETY


1

2

3

4

5

6

7

3.73

1.63

3.54

1.73

Before After

Like

rt 7

Sca

le

15.05 % Reduction in mean value average

FACTOR 4 – LEARNABILITY

21.14% Improvement in Average of Mean value.


1

2

3

4

5

6

7

3.27

1.44

4.75

1.94

Before After

Like

rt 7

Sca

le

FACTOR 5 – ACCEPTABILITY

9.5% Improvement in average of mean value.


1

2

3

4

5

6

7

2.57

1.31

3.24

1.60

Before After

Like

rt 7

Sca

le

FACTOR 6 – EASINESS TO NAVIGATE AND ORIENTATE


1

2

3

4

5

6

7

3.67

1.54

5.21

1.62

Before After

22 % Increase in average of mean value

Like

rt 7

Sca

le

FACTOR 7 – USER CONTROL DESIRE OVER ENVIRONMENT


1

2

3

4

5

6

7

3.12

1.47

4.32

1.32

Before After

17.14 % Increase in average of mean value

Like

rt 7

Sca

le

FACTOR 8 – NATURALNESS AND MEMORABILITY


1

2

3

4

5

6

7

3.02

1.13

4.10

1.25

Before After


Like

rt 7

Sca

le

FACTOR 9 – ACCESSIBILITY


1

2

3

4

5

6

7

2.60

1.13

3.14

1.14

Before After


Like

rt 7

Sca

le

FACTOR 10 – SATISFACTION


1

2

3

4

5

6

7

3.98

1.51

4.76

1.81

Before After


Like

rt 7

Sca

le

FACTOR 11 – EFFICIENCY


1

2

3

4

5

6

7

3.50

1.57

5.42

1.69

Before After


Like

rt 7

Sca

le

FACTOR 12 – EFFECTIVENESS

27.85% improvement in average of mean value


1

2

3

4

5

6

7

3.23

1.10

5.18

1.44

Before After

Like

rt 7

Sca

le

FACTOR 13 – INTUITIVENESS

10.28% improvement in average of mean value


1

2

3

4

5

6

7

4.62

1.60

5.34

1.25

Before After

Like

rt 7

Sca

le

USABILITY

Usability

Sub-FactorMean Value Standard

DeviationBefore After Before After

Learnability 3.27 4.75 1.44 1.94


3.67 5.21 1.54 1.62

Naturalness and memorability

3.02 4.10 1.13 1.25

Average 3.32 4.68 1.37 1.6

Before After01234567

3.32

1.37

4.68

1.6

Mean Value Standard Deviation

19.42% improvement in average of mean value and more usable gesture recognition.

Like

rt 7

Sca

le

USER COOPERATION

User Cooperation

Sub-FactorMean Value Standard

DeviationBefore After Before After

User Control Desire over environment

- 4.92 - 1.76

Accessibility 2.57 3.24 1.31 1.6Satisfaction 3.98 4.76 1.51 1.81

Efficiency 3.5 5.42 1.57 1.69Effectiveness 3.23 5.18 1.10 1.44

Average 3.32 4.704 1.37 1.66

19.77% improvement in average of mean

Before After0

1

2

3

4

5

6

7

3.32

1.3725

4.704

1.66

Mean Value Standard Deviation

Like

rt 7

Sca

le

ALL FACTORS

Factor Before AfterAbsolute value of change

Percentage

Effectiveness 3.23 5.18 1.95 27.86

Efficiency 3.5 5.42 1.92 27.43


3.67 5.21 1.54 22.00

Learnability 3.27 4.75 1.48 21.14

User Control Desire over environment

3.12 4.32 1.2 17.14

Naturalness and memorability 3.02 4.1 1.08 15.43

Satisfaction 3.98 4.76 0.78 11.14

Intuitiveness 4.62 5.34 0.72 10.29

Acceptability 2.57 3.24 0.67 9.57

Accessibility 2.59 3.14 0.55 7.86

Fatigue 3.18 2.66 0.52 7.43

Ergonomic and anxiety 3.73 3.54 0.19 2.71

CONTRIBUTIONS Wireless interactive gesture recognizer

device

Gesture design contribution.

High speed in tracking.

High transmission speed.

Designed Taxonomy and Framework.

CONCLUSION

Qualitative factors general optimization percentage average: 14%

Quantitative factors general optimization percentage average: 19.91%

THANKYOU

Work completion seminar defence

Presentations & Public Speaking

Transcript of Work completion seminar defence