It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 1/35

Lan Zhang, Xiang-Yang Li, Wenchao Huang, Kebin Liu, Shuwei Zong, Xuesi Jian, Puchun Feng, Taeho Jung,

Yunhao Liu ACM. New York, NY, USA ©2014

Advanced Topics on Internet Of Things

Content

1. Background

2. iGaze Idea

3. Challenges and Solutions

4. System Design

5. Empirements and Evaluation

6. Future Work

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 2/35

Background

Collect information sensors

Can run Mobile App

Control, or retrieve data from, other

Support wireless technologies Bluetooth, WiFi, and GPS

May also have all the features of a smartphone

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 3/35

Smart Glasses

Scanning Distance < 3Scanning Angle <30°

High Computation Cost

Displacement > 20cm

QR code

Image Recognition

Hand gesture

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 4/35

Background Existing Technologies

Voice Commands Head Gesture Control TouchPad or Buttons

Smart glasses are still looking for revolutionary applications

Background Human Computer Interface

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 5/35

New Networking Mode

.

Can understand the user's visual attention

using an eye camera

Camera Can automatically connect to the

target of interest

Can run on top of existing networking

protocols, e.g., Wi-Fi.

iGaze Idea

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 6/35

.

Museum

Obtain the detailed descriptions of art works by glazing at it instead.

iGaze Idea

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 7/35

.

A house can turn on a smart appliance only by taking a glance at them

Device ControliGaze Idea

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 8/35

.

When people look at a signboard or product, information can be sent to the potential customer immediately.

Advertising

iGaze Idea

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 9/35

.

iGaze Idea

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 10/35

iGaze Principales

Unidirectional

Given an observer and his/her gaze vector, only the device vector to the correct visual target’s device is consistent with the gaze vector.

iGaze Idea

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 11/35

Bidirectional

Given a pair of users who are

looking at each other, their gaze

vectors have opposite directions.

iGaze Idea iGaze Principales

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 12/35

iGaze Idea

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 13/35

• Develop a low-cost glasses hardwarembedded

with attention capture software

• Capture the gaze direction using the orientation

sensor and eye camera.

Question 1 : How to accurately capture the vision plane attention of a user in real time ?

Answer:

Challenges and Solutions

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 14/35

Visual Attention Detection

• Visual attention detection Fixation

• A fixation duration window threshold for visual attention

Challenges and Solutions

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 15/35

Projection on the image plane is elliptical

Look ahead: the projection looks more like a circle

Look off to one side: it looks more close to an

ellipse

estimate the pose of the iris circle by back-

projecting the ellipse onto a circle in 3D space

Determine Gaze Direction Challenges and Solutions

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 16/35

Capture the direction between glasses

Leveraging Doppler effect caused by arbitrary mild movement of head-

mounted speakers.

Question 2 : How to match the target on device plane according to user’s visual attention?

Answer :

Challenges and Solutions

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 17/35

Challenges and Solutions

Head Node or Shake

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 18/35

Determine Device Direction

System Design

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 19/35

System Design Gaze Vector Acquisition

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 20/35

System Design Device Vector Estimation

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 21/35

System Design Matching and Networking

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 22/35

System Design Application Layer

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 23/35

System DesignVisual Attention Networking (VAN) Protocol

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 24/35

System DesignVisual Attention Networking (VAN) Protocol

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 25/35

System DesignVisual Attention Networking (VAN) Protocol

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 26/35

System DesignVisual Attention Networking (VAN) Protocol

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 27/35

Empirements and Evaluation

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 28/35

Attention & Gaze Direction Accuracy

Attention capture accuracy: ~95%Gaze direction accuracy: < 5° (91 %) < 10° (99 %)

Empirements and Evaluation

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 29/35

Direction estimation is robust against different head gestures and patterns.

Only one mild head gesture is sufficient for highly accurate estimation.

Empirements and Evaluation Device Direction Accuracy

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 30/35

When the distance gets larger, the accuracy decreases

Device Direction AccuracyEmpirements and Evaluation

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 32/35

Computation Cost Energy Cost

Computation Cost & Energy ConsumptionEmpirements and Evaluation

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 33/35

Multiple objects in the gaze direction.

Multiple users may be interested in a same object.

User may be interested in a moving object.

How to avoid insecure connection.

Aesthetic design.

Future Work

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 34/35

It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 35/35

Thank you for listening