www.rrc.mae.ntu.edu.sg

Robotics Research Centre

Funding Agency : Agency for Science, Technology and Research (A*STAR)

Project Motivation & Objectives

Methodology

Results

Principal Investigator:

Prof Gerald Seet

Team Members:

Dr Iastrebov Viatcheslav

Mr Pang Wee Ching,

Mr Dinh Quang Huy

Efficient industrial robot’s detection, identification, tracking and programming by hand-pointing demonstration can

be solved by integrating multiple sensory modalities such as haptic input, gesture input, laser pointing, range data,

visual commendation and task simulation. The goal is to create a partnership dialogue between a human and robot.

This is a one-handed multimodal remote controller for industrial robots. All 5

fingers are ergonomically interfaced to the buttons and motions of the hand

are measured by IMU. Slide haptic input sensor is for smooth fine-tuning

control. Involving of only one hand in the constraint industrial environment

provides a self-support with the remaining free hand. All controls can be

conducted without looking at the device – just by fingers and hand

sensations. Laser pointer, and range-LIDAR are used for Programming By

Demonstration (PBD).

A custom-designed laser outlining projection system has been

implemented as part of HRI. It serves the purpose of generation of

graphics for augmented reality applications. Users, interested in

monitoring the tasks being executed by robot, may experience

difficulties using transparent LCD HMD-glasses in the outside area.

Without the wearable displays, visual interaction between human and

robot would only be possible to share in the notifications by the robot

executed through the “Laser Writer”, which have been mounted on the

wheeled robot. Bright (even at daylight) real time display of comments,

intentions and predictions of the robot can partially replace the AR in

HMD. Whilst the unfilled line drawings may be considered a

disadvantage, the ability to project images in a bright outdoor

environment is an advantage.

1) Haptic and Gesture User Interface (HGIF) Controller

2) Use of lasers in the generation of graphics for augmented

reality applications

3) Framework of Human & Robot Partner Interactions

This is a functional framework for interaction of haptic and hand

motion PBD and AR visual feedback with the Robot. This is not an

architecture, it is a level of abstraction above the architecture. This

framework describes information required by components and for

data flowing among components. Key point here is the decision of

the human: which mode of interaction and control he wants at the

moment: a direct control mode, or assisted mode in which a task

execution scenario is being presented to the Human’s approval,

correction or cancellation.

a. Human-robot partnership can occur due to the dialog between parties: the human must

be able to define and refine his needs for the robot to deliberate and formulate

alternative options for the human.

b. All haptic controls can be conducted without looking at the device – just fingers’ and

hand’s sensations are involved. Feedback is displayed in the HMD glasses and via the

Laser graphics.

c. Laser pointer, and range-LIDAR are used for PBD and are easily activated by haptic or

gesture of one hand.

HGIF Controller