1/69 Manuel Ferre Univ. Politécnica de Madrid March 2006 Human Interfaces for Teleoperation.

1/69

Manuel FerreUniv. Politécnica de Madrid March 2006

Human Interfaces for Human Interfaces for TeleoperationTeleoperation

2/69

OutlineOutline

Haptic interfacesHaptic interfaces

Guidance of telemanipulatorsGuidance of telemanipulators

Stereoscopic visual interfacesStereoscopic visual interfaces

3/69

General control schemeGeneral control scheme

Perception-reasoning-actuationPerception-reasoning-actuation

4/69

Haptic interfacesHaptic interfaces

5/69

Haptics : tactile and kinaesthesia Haptics : tactile and kinaesthesia

Touch is the first information lost in remote Touch is the first information lost in remote telemanipulationtelemanipulation

There are two components involved in manipulation, they There are two components involved in manipulation, they are defined as somatic senses because they are in the are defined as somatic senses because they are in the body:body:

TactileTactile: information is received by the skin and other : information is received by the skin and other closer tissues. Temperature, vibrations and pressure closer tissues. Temperature, vibrations and pressure are detected by these kind of receptors.are detected by these kind of receptors.

Kinaesthesia: Kinaesthesia: information is provided by internal information is provided by internal organs of the body such us muscles and joints. organs of the body such us muscles and joints. Movement and weight of our body is informed by Movement and weight of our body is informed by these organs.these organs.

Haptic perception is the combination of both impressions Haptic perception is the combination of both impressions

6/69

Haptics : tactile and kinaesthesiaHaptics : tactile and kinaesthesia

Main features of haptics are:Main features of haptics are:

TactileTactile: : Detects temperature and Detects temperature and

patterns of pressurepatterns of pressure

High frequency stimulusHigh frequency stimulus

Distributed on all skinDistributed on all skin

KinaesthesiaKinaesthesia:: Detects body positions and Detects body positions and

movementsmovements

Low frequency stimulusLow frequency stimulus

Related to muscular effort Related to muscular effort

7/69

Haptics : physiology of touchHaptics : physiology of touch

Main cutaneous receptors:Main cutaneous receptors:

8/69

Haptics : physiology of touchHaptics : physiology of touch

Minimum value able to Minimum value able to perceive:perceive:

• 80 mg in the finger tip80 mg in the finger tip• 150 mg in the palm150 mg in the palm

Higher sensitivity at 32º C, from Higher sensitivity at 32º C, from 40º C stimulus is degraded40º C stimulus is degraded

Higher surface in the contact Higher surface in the contact implies higher sensitivity when implies higher sensitivity when pressure is changedpressure is changed

9/69

Haptics : stimulus resolutionHaptics : stimulus resolution

Resolution of touch:Resolution of touch:

Spatial resolution varies between 1-2 Spatial resolution varies between 1-2 mmmm22 to 45 cmto 45 cm22 according to the location of the according to the location of the stimulus:stimulus:

• FingerFinger: minimum distance to distinguish two : minimum distance to distinguish two stimuli is 2.5 mmstimuli is 2.5 mm

• PalmPalm: minimum distance is about 10 mm: minimum distance is about 10 mm

• ThighThigh: distance of less than 67 mm is difficult to : distance of less than 67 mm is difficult to distinguishdistinguish

Temporal resolution: its minimum value is Temporal resolution: its minimum value is around 5 ms.around 5 ms.

10/69

Haptics : stimulus resolutionHaptics : stimulus resolution

Sensitivity of haptic stimuli is Sensitivity of haptic stimuli is measured by the measured by the JND (Just JND (Just Noticeable Difference)Noticeable Difference)

11/69

Haptics : graspingHaptics : graspingGraspingGrasping

Forces depends on: geometry, Forces depends on: geometry, constitution, skill.constitution, skill.

Two different kind of grasping:Two different kind of grasping:• Power:Power:

–High forces and stable configurationHigh forces and stable configuration–Palm and fingers are usedPalm and fingers are used–Maximum applied force is around 400 N Maximum applied force is around 400 N

(men) and 228 N (women)(men) and 228 N (women)• Precision:Precision:

–Low force is exertedLow force is exerted–Finger are skilfully usedFinger are skilfully used

12/69

Haptics : graspingHaptics : grasping

Classification of graspingClassification of grasping

13/69

Haptics : bandwidth Haptics : bandwidth

Bandwidth of haptic stimuli:Bandwidth of haptic stimuli:Each receptor has a specific Each receptor has a specific

range:range:• Vibration: 5-10 kHz (texture)Vibration: 5-10 kHz (texture)

• Tactile: 0-400 HzTactile: 0-400 Hz

• Kinaesthesia: 20-30 HzKinaesthesia: 20-30 Hz

The human answer to the haptic The human answer to the haptic stimuli (our actions) is around stimuli (our actions) is around 5 – 10 Hz5 – 10 Hz

14/69

Haptics : bandwidthHaptics : bandwidth

Finger bandwidthFinger bandwidth

15/69

Haptics : bandwidthHaptics : bandwidth

Finger bandwidthFinger bandwidth

16/69

Haptics : requirementsHaptics : requirements

Main requirements for haptics interfacesMain requirements for haptics interfaces

17/69

Haptics : devicesHaptics : devices

Background:Background:

1940s1940s: Tongs for manipulating hazardous : Tongs for manipulating hazardous materialsmaterials

18/69


GRIPS telemanipulator by Kraft Telerobotics:GRIPS telemanipulator by Kraft Telerobotics:Electro-HidraulycElectro-HidraulycBilateral master-slaveBilateral master-slave6 DOF + grip6 DOF + grip

19/69


TITAN by Schilling:TITAN by Schilling:

Electro-HidraulycElectro-Hidraulyc

Bilateral master-slaveBilateral master-slave

Force sensor in the wristForce sensor in the wrist

6 DOF + grip6 DOF + grip

20/69


Phantom by Phantom by SensAble Tech.:SensAble Tech.:

Force reflected in Force reflected in 3 dof3 dof

Serial Serial configuration configuration

Large bandwidth Large bandwidth (pulley (pulley transmission)transmission)

21/69


PUMA robot and PUMA robot and master interface by master interface by Oxford UniversityOxford University

Master is a 6 dof Master is a 6 dof parallel device parallel device (Steward platform)(Steward platform)

Large bandwidthLarge bandwidth

Indexed workspaceIndexed workspace

22/69


Magister parallel haptic interfaced Magister parallel haptic interfaced developed at UMH. developed at UMH.

23/69


Joysticks and similar devices:Joysticks and similar devices:

Rate controlRate control

Low workspaceLow workspace

Focused on specific developmentsFocused on specific developments

24/69


Hands developed by Hands developed by K. Salisbury at the K. Salisbury at the MIT:MIT:

16 gdl16 gdl

Simplification of Simplification of human handhuman hand

Exos Dextrous Hand Exos Dextrous Hand Master:Master:

16 gdl16 gdl

25/69


MIT hand and Exos Dextrous Hand MIT hand and Exos Dextrous Hand Master integratedMaster integrated

26/69


Artificial hand developed by CSIC-IAI:Artificial hand developed by CSIC-IAI:

3 fingers actuated and 2 fixed3 fingers actuated and 2 fixed

27/69


Exoesqueleton of Utah UniversityExoesqueleton of Utah University

28/69


Space-Mouse developed by G. Hirzinger:Space-Mouse developed by G. Hirzinger:

6 gdl6 gdl

Forces and torques are measuredForces and torques are measured

29/69


Gloves:Gloves:

Hand posture is identifiedHand posture is identified

Resolution is poorResolution is poor

30/69

GuidanceGuidance

31/69

GuidanceGuidance

Main topics:Main topics:

Time executionTime execution

TeleproprioceptionTeleproprioception

Working areaWorking area

32/69

Guidance Guidance

Operator’s movements are transformed to Operator’s movements are transformed to movement commands to remote robotsmovement commands to remote robots

33/69

GuidanceGuidance : time execution: time execution

Time execution:Time execution:

This is an experimental parameter that shows This is an experimental parameter that shows how easy is executed a task of guidancehow easy is executed a task of guidance

It permits to compare different devices and It permits to compare different devices and technologiestechnologies

Productivity is connected to time executionProductivity is connected to time execution

Learning is strongly Learning is strongly connected connected to time execution to time execution in the firsts in the firsts trialstrials

0

10

20

30

40

50

60

0 10 20 30 40 50

Task nº

Tim

e (m

in)

34/69

GuidanceGuidance : teleproprioception: teleproprioception

Propioception: Propioception: It is the ability of a human being to know the It is the ability of a human being to know the

status of his membersstatus of his membersThis information is obtained by the body This information is obtained by the body

joints, ear and other nerve impulsesjoints, ear and other nerve impulses

Telepropioception:Telepropioception:It is the ability of a human operator to know It is the ability of a human operator to know

the consequences of his actions with respect the consequences of his actions with respect to the remote environment.to the remote environment.

It implies that remote and local reference It implies that remote and local reference systems must be coherentsystems must be coherent

35/69

GuidanceGuidance : teleproprioception: teleproprioception

Disruption between stimuli and actions:Disruption between stimuli and actions:

Mechanical disruption Mechanical disruption

Geometrical disruption Geometrical disruption

C om m anded m otion

Expected m otion

O bserved m otion

Actualm otion

Com m anded m otion

Expected m otion

O bserved m otion

Actualm otion

36/69

Guidance : indexationGuidance : indexation

Working areaWorking area

It happens when slave workspace is It happens when slave workspace is bigger than master. Examples:bigger than master. Examples:

Big slave and small masterBig slave and small master

Different kinematicsDifferent kinematics

SolutionSolution: indexation (telepropioception is lost): indexation (telepropioception is lost)

37/69

Conclusions about haptics and guidance Conclusions about haptics and guidance

Bilateral devices must guarantee Bilateral devices must guarantee operator’s teleproprioceptionoperator’s teleproprioception

Kinaesthetic stimulus is predominant in Kinaesthetic stimulus is predominant in force reflectionforce reflection

Tactile stimulus is increased according Tactile stimulus is increased according to the bandwidth of the haptic interfaceto the bandwidth of the haptic interface

Low bandwidth is useful for detect Low bandwidth is useful for detect collision. Is it sufficient? …. It depends collision. Is it sufficient? …. It depends on applicationon application

38/69

Visual interfacesVisual interfaces

39/69

Visual interfaces: outlineVisual interfaces: outline

Description of visionDescription of vision

Spatial perceptionSpatial perception

DevicesDevices

40/69

Visual interfaces: visionVisual interfaces: vision

Vision: is the sense that detects light. ItVision: is the sense that detects light. It informs informs about shape, size, movements, distances and colour.about shape, size, movements, distances and colour.

This stimulus is caused by the light received throughThis stimulus is caused by the light received through the retina photoreceptors.the retina photoreceptors.

Crystalline is a lens that changes its shape to focus Crystalline is a lens that changes its shape to focus light on near or far objects.light on near or far objects.

41/69

Visual interfaces: visionVisual interfaces: visionThere are two types of light sensors in the There are two types of light sensors in the

retina:retina:

Rods: work best in dim light and allow black Rods: work best in dim light and allow black and white vision and white vision

Cones: work best in bright light and detect Cones: work best in bright light and detect color. In the human eye, cones respond to red, color. In the human eye, cones respond to red, green and blue wavelengths of light green and blue wavelengths of light (29-70).(29-70).

42/69


Eye resolution is due to Eye resolution is due to light sensors size. light sensors size. Maximum resolution is between Maximum resolution is between 30 sec. arc. and 1 30 sec. arc. and 1 min. It depends on light conditions.min. It depends on light conditions.

Resolution decreases from fovea to periphery. Resolution decreases from fovea to periphery. Sensor resolution is a limit for pixel size Sensor resolution is a limit for pixel size

43/69


The image frequency perceived depends on The image frequency perceived depends on illumination conditions. This frequency is close illumination conditions. This frequency is close to 20-25 images per second. Lower frequencies to 20-25 images per second. Lower frequencies produces flicking.produces flicking.

The visible spectrum is roughly from 700 The visible spectrum is roughly from 700 nm (red end) to 400 nm (violet end).nm (red end) to 400 nm (violet end).

44/69

Visual interfaces: depth visionVisual interfaces: depth vision

Field of view is the part of the observable Field of view is the part of the observable world that is seen. Humans have a world that is seen. Humans have a 180º180ºxx120º FoV.120º FoV.

Binocular vision only covers 120º degrees Binocular vision only covers 120º degrees of the vision field in humans. The of the vision field in humans. The remaining area is the peripheral vision remaining area is the peripheral vision

45/69

Visual interfaces: depth visionVisual interfaces: depth vision

Depth perception combines several types Depth perception combines several types of depth clues grouped into two categoriesof depth clues grouped into two categories

Monocular clues:Monocular clues: Motion parallaxMotion parallax

Image realism (Color vision, perspective, relative Image realism (Color vision, perspective, relative size, occlusion, texture gradient)size, occlusion, texture gradient)

Binocular clues:Binocular clues: Stereopsis Stereopsis

ConvergenceConvergence

46/69

Visual interfaces: depth perceptionVisual interfaces: depth perception

Motion parallax : Motion parallax :

The apparent relative motion of The apparent relative motion of several stationary objects against a several stationary objects against a background when the observer is background when the observer is moving. moving.

This effect can be seen clearly This effect can be seen clearly when driving a car, near things pass when driving a car, near things pass quicker, while far off objects appear quicker, while far off objects appear stationary. stationary.

This mechanism is predominant This mechanism is predominant for far objects (>1m)for far objects (>1m)

47/69

Visual interfaces: depth perceptionVisual interfaces: depth perceptionImage realism is a monocular clue that provides spatial Image realism is a monocular clue that provides spatial

information according to our experience (pictures, movies)information according to our experience (pictures, movies)

Color visionColor vision: correct interpretation of color, and especially : correct interpretation of color, and especially lighting clues, allow the beholder to determine the object lighting clues, allow the beholder to determine the object shape.shape.

PerspectivePerspective: the feature of parallel lines converging at infinity : the feature of parallel lines converging at infinity allows us to reconstruct relative distances.allows us to reconstruct relative distances.

Relative sizeRelative size: close objects to us seem larger than further : close objects to us seem larger than further ones; our visual system exploits the relative size of known ones; our visual system exploits the relative size of known objects to judge distancesobjects to judge distances

OcclusionOcclusion: close objects blocking the sight of far objects, this : close objects blocking the sight of far objects, this clue informs us about relative distances.clue informs us about relative distances.

48/69


Several levels of realism:Several levels of realism:

49/69


Binocular disparity:Binocular disparity:

By using two images of the same By using two images of the same object taken from slightly different object taken from slightly different points of view, it is possible to know points of view, it is possible to know the distance to an object.the distance to an object.

Binocular disparity resolution:Binocular disparity resolution: Resolution 1 min.Resolution 1 min.

Eye distance 6,5 cm.Eye distance 6,5 cm.

46 cm from object to eye implies a 46 cm from object to eye implies a resolution of 0,5 mmresolution of 0,5 mm

It is predominant in close distance It is predominant in close distance (<1m).(<1m). 6.5 cm6.5 cm

46 cm46 cm

0.5 cm0.5 cm

50/69


Convergence:Convergence: it represents the angle between eyes it represents the angle between eyes when they are fixed to an object. Convergence when they are fixed to an object. Convergence angle is larger when objects are closer.angle is larger when objects are closer.

StereopsisStereopsis: : is the ability to blend images of is the ability to blend images of both eyes together, it allows us to both eyes together, it allows us to appreciate depth and judge distances.appreciate depth and judge distances.

51/69


Stereopsis depends on the slight Stereopsis depends on the slight differences (disparity) in the two pictures differences (disparity) in the two pictures projected on the retinas. The brain can fuse projected on the retinas. The brain can fuse both images when disparity is into the both images when disparity is into the proper limits.proper limits.

52/69


Binocular disparity is the horizontal Binocular disparity is the horizontal distance between related points in both distance between related points in both images. It depends on:images. It depends on:

Distance between object to camera, andDistance between object to camera, and

Convergence angle of camerasConvergence angle of cameras

53/69


Un-crossed disparity Zero disparity Crossed disparity

Three possible Three possible values of values of disparitydisparity

54/69


0 20 40 60 80 100 120 140 160 180 2000

5

10

15

20

25

30

35

40

45

50

ConvergenceConvergence 4040 6060 8080

Range (cm.)Range (cm.) 23-7023-70 40-11040-110 50-15050-150

x

dxoabsdisparidad

55/69


Brain fusion is limited by the maximum Brain fusion is limited by the maximum value of disparityvalue of disparity

It is a limit to use stereoscopic images in It is a limit to use stereoscopic images in telemanipulationtelemanipulation

0

1

2

3

4

5

6

7

8

9

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 115 120 125 130 135 140 145 150 155 160 165 170 175 180

DISTANCE BETWEEN CAMERA AND OBJECT

IMAGEDISPARITY

NO FUSED IMAGESFUSEDIMAGESS

56/69

Visual interfaces: experimentsVisual interfaces: experiments

Experiment about limits of disparityExperiment about limits of disparity

Setup:Setup:

Steroscopic images are recordedSteroscopic images are recorded3 different convergences and 10 distances 3 different convergences and 10 distances

object-cameraobject-camera

20 participants20 participants

0

1

2

3

4

5

6

7

8

9

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 115 120 125 130 135 140 145 150 155 160 165 170 175 180

DISTANCE BETWEEN CAMERA AND OBJECT

IMAGEDISPARITY

NO FUSED IMAGESFUSEDIMAGESS

57/69


Maximum disparity around to:Maximum disparity around to: 2º 2º

0

2

4

6

8

10

12

Distancia de la cámara al objeto (cm)

Dis

pari

dad

an

gu

lar

y V

alo

ració

n

este

reo

scó

pic

a

Disparidad Angular Valoración estereoscópica

58/69


Experiment to compare stereo vs. Experiment to compare stereo vs. Monoscopic images for teleroboticsMonoscopic images for telerobotics

Experiment to compare stereo vs. Experiment to compare stereo vs. Monoscopic images for teleroboticsMonoscopic images for telerobotics

59/69


Comparison between mono and stereo Comparison between mono and stereo imagesimages

0

50

100

150

200

250

300

350

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

Operators

Exe

cuti

on T

ime

(s)

Estereo

Mono

60/69

Visual interfaces: devicesVisual interfaces: devices

61/69


Stereoscopic and y Stereoscopic and y autoestereoscopicautoestereoscopic

Stereoscopic Stereoscopic : they use : they use additional material like additional material like glasses or a helmet to glasses or a helmet to show a different image to show a different image to each eye.each eye.

Autostereoscopic: they Autostereoscopic: they show a different image to show a different image to each eye without each eye without additional devices.additional devices.

62/69


Binocular devicesBinocular devices

ParallelParallel StereoscopeStereoscope

Head Mounted Head Mounted Displays (HMD)Displays (HMD)

Polarization coding Polarization coding glasses (IMAX)glasses (IMAX)

Color coding glassesColor coding glasses

SequentialSequentialActive shutterActive shutter

63/69

Visual interfaces: devicesVisual interfaces: devicesAutostereoscopic:Autostereoscopic:

Lenticular sheets. Lenticular sheets. This device is based on an array of This device is based on an array of cylindrical lenses that generates a threedimensional image by cylindrical lenses that generates a threedimensional image by showing different vertical sections to each eye. showing different vertical sections to each eye.

The observer’s head must be placed at a predefined position The observer’s head must be placed at a predefined position in order to see the 3D image properly, otherwise the cross in order to see the 3D image properly, otherwise the cross sections will be overlaid as the head moves.sections will be overlaid as the head moves.

64/69


Parallax barrier.Parallax barrier. This is a vertical slit plate placed in This is a vertical slit plate placed in front of a display in order to block part of the screen front of a display in order to block part of the screen shown to each eye. The screen displays two images shown to each eye. The screen displays two images divided into vertical strips. Each eye can only see the divided into vertical strips. Each eye can only see the corresponding strips due to the slit plate performance.corresponding strips due to the slit plate performance.

The observer’s eyes must be placed at the proper The observer’s eyes must be placed at the proper position to avoid the blending of the strips.position to avoid the blending of the strips.

65/69


Holograms and autostereogramsHolograms and autostereograms

66/69


Immersive system:Immersive system:

Curve-screen theatreCurve-screen theatre

Flat-screen walls.Flat-screen walls.

67/69

Visual interfaces: examplesVisual interfaces: examples

3D interface of ROTEX3D interface of ROTEX

68/69

Visual interfaces: examplesVisual interfaces: examples

TelesurgeryTelesurgery

NASANASA DeminingDemining

69/69

Visual interfaces: conclusionsVisual interfaces: conclusions

Stereo vision is necessary for proper Stereo vision is necessary for proper guidance in order to avoid uncertainty guidance in order to avoid uncertainty

Remote workspace is limited by maximum Remote workspace is limited by maximum disparity (2º). Active convergence is a disparity (2º). Active convergence is a solutionsolution

Excellent quality of images are required Excellent quality of images are required (monocular clues). It implies specific (monocular clues). It implies specific hardwarehardware

Many solutions are in the market, none of Many solutions are in the market, none of them can reproduce human vision them can reproduce human vision perceptionperception

1/69 Manuel Ferre Univ. Politécnica de Madrid March 2006 Human Interfaces for Teleoperation.

Documents

Transcript of 1/69 Manuel Ferre Univ. Politécnica de Madrid March 2006 Human Interfaces for Teleoperation.