2. Visually- Guided Grasping (3D) - Universitat Jaume Isanzp/irs/AutRobManipula-2011-3.pdf · April...
Transcript of 2. Visually- Guided Grasping (3D) - Universitat Jaume Isanzp/irs/AutRobManipula-2011-3.pdf · April...
5/9/2011
1
Autonomous Robotic Manipulation (3/4)
Pedro J Sanz
April 2010 Fundamentals of Robotics (UdG) 2
2. Visually-
Guided
Grasping (3D)
5/9/2011
2
April 2010 Fundamentals of Robotics (UdG) 3
Analyzing complete 3D object model
3D reconstruction
CAD model
Object Recognition
Extensive training sessions Previously
known!
Off-line Time
Consuming
Complex and
Non-reactive
algorithms
• Other approaches for finding 3D grasps
April 2010 Fundamentals of Robotics (UdG) 4
Towards 3D Visually-Guided Grasping with a Dexterous Hand
Grasping of previously unknown 3D objects using
visual information (camera in hand)
2D features from contour
multiple views
as few views as possible
Model objects with a bounding box
3D Partial
Reconstruction
5/9/2011
3
April 2010 Fundamentals of Robotics (UdG) 5 X
Y
Z
Start
Robot Base
Coordinate System
Top1
Top2
follow centroid
length
height
width
front position
Z-Offset
Work area
(known)
Center of Mass /
Center of Bounding Box
dx
dz
distance
• Our Method
April 2010 Fundamentals of Robotics (UdG) 6
• Grasping Procedure
video-06
5/9/2011
4
April 2010 Fundamentals of Robotics (UdG) 7
• Height refinement through front view
April 2010 Fundamentals of Robotics (UdG) 8
Partial model
per View
3-D Object Model
based on geometric
features
Contour
Analysis
e.g. centroid, curvature, Imin, Imax, bounding box, orientation, …
Object
Contour
Grasp Planning
Internal 2-D
Models
(Views)
Views +center of mass, +center of bounding box +length, width, height
3-D Analysis next position
• Object Model Generation
5/9/2011
5
April 2010 Fundamentals of Robotics (UdG) 9
Grey Image Binarisation/
Segmentation
Contour
Extraction Undistortion
Object Model
Generation
• Image Processing
April 2010 Fundamentals of Robotics (UdG) 10
x
y
Imin
Binarized Image
Imax
(xc,yc)
Image I with N pixels
Object O
Moment of order p+q
x y
qp
pq yxIyxM ,
00
01
00
10 ,,M
M
M
MyxC cc
yxI ,1 if (x,y) O
0 else where
Centroid
x y
q
c
p
cpq yxIyyxxM
µ ,1
00
Normalized central moment of order p+q
200211,*2atan2*5.0
cos
sinminI
sin
cosmaxI
Orientation of Imin towards y-axes
Axis of Inertia
• Centroid and Axis of Inertia
5/9/2011
6
April 2010 Fundamentals of Robotics (UdG) 11
• Following the Centroid
video-05
April 2010 Fundamentals of Robotics (UdG) 12
Center of
Image
dy
dx
img
timg
tooltool
t vRv
*
0
* dy
dx
v img
t
x
y
x
y
Image Coordinate System
Tool Coordinate
System
2
100
02
cos2
sin
02
sin2
cos
Roll
π
Rimg
tool
Velocity Control with Visual Servoing 2-D
• Following the Centroid
5/9/2011
7
April 2010 Fundamentals of Robotics (UdG) 13
Pose1
y x
100
0
tan
yp
f
xp
f
p
f
c
c
Cy
yx
1
*
1
1 y
x
CY
X
ray
Pose2
y x
1ray
2ray
P
Corresponding
Points
Camera calibration matrix:
From image points to rays
Rays in general skewed!
• 3-D Reconstruction
14
• Current Experimental Setup
Mitsubishi Robot Arm (PA-10)
Three-Fingered Hand (BarrettHand)
Camera in Hand (Sony XC-333)
Kinematics
5/9/2011
8
3D-Object
Model
Grasp
Synthesis
Grasp
Control
Grasping type (precision / power), desired Tool pose and hand pre-shape (spread)
Approaching Grasping
Grasp
Characterization
Feasible grasp regions
Grasp regions (GR) ->compatible GR ->feasible GR
Filtering for executable grasps (size) Choosing best grasp (manually) Generating pose and spread of hand
• Grasp Planning
April 2010 Fundamentals of Robotics (UdG) 16
2 Finger Grasps 3 Finger Grasps
• Grasp Synthesis from Top
5/9/2011
9
April 2010 Fundamentals of Robotics (UdG) 17
• Grasp Synthesis from Front
April 2010 Fundamentals of Robotics (UdG) 18
12
thresh 21
• Grasp Synthesis from Front
5/9/2011
10
April 2010 Fundamentals of Robotics (UdG) 19
Examples of top grasps
Examples of front grasps
5/9/2011
11
April 2010 Fundamentals of Robotics (UdG) 21
Characteristics of Top Grasp
• Vertical Position
• Orientated and Positioned with respect to
Grasping Regions
• Precision Grasp
– Spread if 3 finger grasp selected
– Finger tips stay at the same z-level
April 2010 Fundamentals of Robotics (UdG) 22
Grasp Execution From Front
• Horizontal Position
• Positioned always at Height of Center of
Mass
• Distance of Length/2 to Center of
Bounding Box
• Power Grasp
– No spread
– Palm stays at object for support
5/9/2011
12
April 2010 Fundamentals of Robotics (UdG) 23
Examples
video-07
April 2010 Fundamentals of Robotics (UdG) 24
Problems?
video-08
5/9/2011
13
April 2010 Fundamentals of Robotics (UdG) 25
Towards Complex Tasks
video-09
April 2010 Fundamentals of Robotics (UdG) 26 SET 2008 26 IROS'08
Conclusions and Future
Lines • A new approach for grasping previously
unknown 3D objects has been
implemented and tested on a real robotic
system
• Force/torque and tactile sensors are
starting to use now for compensating the
inaccuracies of both, grasp planning and
visual perception
5/9/2011
14
3. Sensor-based Control Interaction
3.1 planning of physical interaction tasks [ICRA-07-Prats]
3.2 vision-force-tactile integration for
robotic physical interaction [ICRA-09-Prats]
Mario Prats
Email: [email protected]
“Robotic Physical Interaction
through the Combination of Vision,
Tactile and Force Feedback”
PhD Thesis (UJI; June 21st 2009) European Doctorate
http://creatures.uji.es/~mprats/thesis/Mario_Prats_Thesis.pdf
April 2010 Fundamentals of Robotics (UdG) 28
Preliminary ideas
5/9/2011
15
April 2010 Fundamentals of Robotics (UdG) 29
What is a Service Robot?
It is an intermediate state between the
Industrial Robot and the Humanoid Robot
Definition-1
April 2010 Fundamentals of Robotics (UdG) 30
Robots working and collaborating with
humans in human environments
Definition-2
• Unstructured and changing environment
• Unknown objects
• Poorly defined tasks
• Inaccuracy in the information about the environment
Properties
Uncertainty
is a main concern
5/9/2011
16
A Brief History about AI Robots
“Shakey” (SRI 1967) vs “Herbert” (MIT 1990)
Hierarchical Paradigm Reactive Paradigm
April 2010 Fundamentals of Robotics (UdG) 32
Hierarchical Paradigm
Environment
SENSE ACT
PLAN
STRIPS [Nilsson, Rosen et al., 1969]
Stanford Research Institute Problem Solver
Task-Planner
“Shakey”
5/9/2011
17
April 2010 Fundamentals of Robotics (UdG) 33
Reactive Paradigm
Environment
SENSE
ACT
Intelligence!
“Herbert”
[Connell, 1990]
Behavior-Based Robotics
Visually-Guided
Grasping
5/9/2011
18
Robot control system spectrum Adapted from [Arkin & Grupen, 93]
SPEED OF RESPONSE
PREDICTIVES CAPABILITIES
DEPENDENCE ON ACCURATE, COMPLETE WORLD MODELS
DELIBERATIVE
Reflex Response
REACTIVE
Purely Symbolic
Representation-free
Real-time response
Low-level intelligence
Representation-dependent
Slower response
High-level intelligence (cognitive)
April 2010 Fundamentals of Robotics (UdG) 36
Which are our expectations
for this kind of
robots?
5/9/2011
19
April 2010 Fundamentals of Robotics (UdG) 37
Human attendance in museums, hospitals, etc.
Maggie
Robotics Lab
UCIII, Madrid
April 2010 Fundamentals of Robotics (UdG) 38
Attending elderly and disabled people
University of Vanderbilt, US
University of Bremen, Germany
ISAC project
FRIEND project
5/9/2011
20
Univ
ersity o
f Karlsru
he, G
erman
y
Are Autonomous Actions Necessary?
April 2010 Fundamentals of Robotics (UdG) 40
3.1 Planning
of physical
interaction
tasks
5/9/2011
21
April 2010 Fundamentals of Robotics (UdG) 41
Preliminary concepts
• the contact-level approach vs the
knowledge-based approach
• Task-oriented grasps and grasp-
oriented tasks
April 2010 Fundamentals of Robotics (UdG) 42
The concept of physical interaction “ … is introduced to refer indistinctly to the grasp
(prehensile and non-prehensile) and to the task”
5/9/2011
22
April 2010 Fundamentals of Robotics (UdG) 43
The Questions to Answer
• How can everyday physical interaction be specified in a
common framework, including both the grasp and the
task, and supporting sensor-based control?
• How can a robot autonomously plan a physical
interaction task, making use of this framework?
• What sensors are necessary, and how can a robot
combine those sensors and control its motors for
performing physical interaction tasks in a robust
manner?
April 2010 Fundamentals of Robotics (UdG) 44
Grasp preshapes and shape primitives The six basic prehensile patterns defined by Schlesinger (1919), adapted from
(Taylor & Schwarz, 1955)
5/9/2011
23
April 2010 Fundamentals of Robotics (UdG) 45
The physical interaction frames
(left): the object frame (O), the end-effector frame (E), the task frame (T), the hand
frame (H) and the grasp frame (G).
The task motion must be transformed into robot coordinates through the kinematics
chain formed by T, G, H and E (right).
The grasp link is the relative pose between frames H and G, and represents the
bridge between the task and the grasp
April 2010 Fundamentals of Robotics (UdG) 46
Sensor-based tracking of the
physical interaction frames
The use of sensor feedback is specially important for the estimation
of the grasp link
5/9/2011
24
April 2010 Fundamentals of Robotics (UdG) 47
Example: Use of tools The use of tools consists of two different instances of physical
interaction tasks: one, involving a grasp for a transport task, and
another one, involving the actual use of the tool
Grasping the tool Reaching for the object Using the tool
April 2010 Fundamentals of Robotics (UdG) 48
Task-oriented hand preshapes The ideal task-oriented hand preshapes considered by
the physical interaction task planner
5/9/2011
25
Mapping to the Barrett Hand The Barrett Hand task-oriented hand preshapes
April 2010 Fundamentals of Robotics (UdG) 50
Mapping to a Parallel Jaw gripper
The parallel jaw gripper task-oriented hand preshapes
5/9/2011
26
April 2010 Fundamentals of Robotics (UdG) 51
Object representation (e.g. Structural model of a cabinet)
April 2010 Fundamentals of Robotics (UdG) 52
Classification of object parts
5/9/2011
27
April 2010 Fundamentals of Robotics (UdG) 53
Task description
April 2010 Fundamentals of Robotics (UdG) 54
Planning Some examples of the grasp frame specification according to the task
description and selected preshapes
5/9/2011
28
April 2010 Fundamentals of Robotics (UdG) 55
3.2 vision-force-
tactile
integration for
robotic physical
interaction
ICRA-09
ICRA-09
The experimental environment consists of a mobile manipulator,
an external camera and a cabinet with a sliding door that must be
pushed open to the left
AN EXPERIMENT
5/9/2011
29
April 2010 Fundamentals of Robotics (UdG) 57
4. The UJI
Service Robot:
A Case Study
April 2010 Fundamentals of Robotics (UdG) 58
International Cooperation
The UMass Torso (USA)
Prof. Grupen
LASMEA
Le Laboratoire Sciences Matériaux
l'Electronique 'Automatique
Prof. Melchiorri
Prof. Martinet
Italy
France
5/9/2011
30
International Cooperation
TECHNISCHE UNIVERSITÄT
MÜNCHEN
Germany Prof Färber
Prof Dillman
Germany
Looking for Books in a Library
University of Tsukuba
“Remote book browsing
system using a mobile
manipulator”
Tomizawa
http://www.roboken.esys.tsukuba.ac.jp/ ICRA’03
Johns Hopkins University “Comprehensive Access to
Printed Materials (CAPM)”
ICRA’02 Choudhury http://dkc.jhu.edu/CAPM/
5/9/2011
31
April 2010 Fundamentals of Robotics (UdG) 61
IROS’06
“The UJI Librarian Robot”
ICRA’05
IROS’05
Looking for Books in a Library
April 2010 Fundamentals of Robotics (UdG) 62
Main Steps
User Input
Scanning Mechanism
Success?
Grasping module
yes
not
1
2
3
4
Labels based on
Library of Congress
Classification (USA)
Label
5/9/2011
32
April 2010 Fundamentals of Robotics (UdG) 63
System Sketch
OCR
3
4
2
1
Identification, Localization and Extraction
5/9/2011
33
April 2010 Fundamentals of Robotics (UdG) 65
The System in Action
[ICRA’05]
1 2
3 4
video-11
April 2010 Fundamentals of Robotics (UdG) 66
Grasping Sketch
Force Feedback
5/9/2011
34
April 2010 Fundamentals of Robotics (UdG) 67
IROS’06
ICRA’05
Evolution of the Project
ICRA’07
April 2010 Fundamentals of Robotics (UdG) 68
From “Jaume” to “Jaume-2”
5/9/2011
35
April 2010 Fundamentals of Robotics (UdG) 69
The Software Architecture
April 2010 Fundamentals of Robotics (UdG) 70
The Task Frame Formalism (TFF)
5/9/2011
36
April 2010 Fundamentals of Robotics (UdG) 71
The Task Frame Formalism (TFF)
Example 1: “the robot turning the door handle”
April 2010 Fundamentals of Robotics (UdG) 72
Example 2: “the UJI Librarian Robot”
5/9/2011
37
April 2010 Fundamentals of Robotics (UdG) 73
“Jaume-2
” i
n A
cti
on
April 2010 Fundamentals of Robotics (UdG) 74
“Jaume-2
” i
n A
cti
on
5/9/2011
38
April 2010 Fundamentals of Robotics (UdG) 75
“Jaume-2
” i
n A
cti
on
April 2010 Fundamentals of Robotics (UdG) 76
Our new platform