Kinematic Model of the Hand using Computer VisionEnglish).pdf · Kinematic hand model solver...

Kinematic Model of the Hand

using Computer Vision

Edgar Simo Serra

Barcelona, Spain

Barcelona - April, 2011

Summary

Obtain hand movements through computervision

Reconstruct hand kinematics from data

1 Overview

2 Kinematic Hand Model

3 Hand Detection

4 Kinematic Synthesis

5 Non-linear solver

6 Results

Overview

Motivation

EMG Signals ⇐⇒ Hand Movement

Mapping of Electromyographic (EMG) signals

Understanding hand movements

Prosthesis design

Exoskeleton design

Objectives

1 Design of a theoretical kinematic model of thehand

2 Detection of real hand characteristics withcomputer vision

3 Solver to adjust the theoretical model to theexperimental data

Kinematic Hand Model

State of the Art

Robotic Grippers

Prosthetic Hands

Match function not form

Simplified kinematic models

State of the Art

Robotic Grippers

Prosthetic Hands

State of the Art

Robotic Grippers

Prosthetic Hands

Hand Anatomy

5 Fingers

14 Joints

27 Bones

Complex articulatedsystem

Hand Topology

3 common joints (wrist)

Two fingers with 4 joints (index, middle)

Three fingers with 5 joints (third, fourth,thumb)

Thumb has different joint structure

Hand Topology

(a) Full hand. (b) Index finger. (c) Middle finger.

(d) Third finger. (e) Fourth finger. (f) Thumb.

Hand Model

26 degrees of freedomAll revolute jointsStrong similarity with bone structure

Detection of Hand Characteristics

State of the Art

Contact approachLow portabilityHigh costNeed for calibrationLow resolution

State of the Art

Contact approachLow portabilityHigh costNeed for calibrationLow resolution

Non-contact ApproachFocus on realtimeLow precisionApplications in augmentedreality

Approach

Tracking of nailsNon-deformableFixed to hand structureEasily identifiableNot easy to identify rotation

Markers for robust designAllows obtaining of pose (position and rotation)

Markers

Simple for robustness

Convey poseinformation

Markers

Simple for robustness

Convey poseinformation

Detecting Markers

1 Detect candidate points with Corner and edgedetector (Harris 1998)

2 Match candidate points with heuristics3 Use DAISY descriptors to do tracking (Tola

Detecting Markers

1 Detect candidate points with Corner and edgedetector (Harris 1998)

2 Match candidate points with heuristics3 Use DAISY descriptors to do tracking (Tola

PnP: Pose Estimation Problem

u = [A][R |t]p

Very important problem in computer vision

Identify camera position given n 2D-3Dcorrespondences u ↔ p

Solving the Pose Estimation

Specific case 4 coplanar points: P4P

Single unique solution (Yuan 1989)Usage of combination of algorithms

Orthogonal Iteration (Hager et al. 2000)Robust Pose Estimation (Schweighofer et al. 2006)

Error Analysis

0 50 100 150 200

Pose (0 degrees)

0 50 100 150

Pose (10 degrees)

0 50 100 150

Pose (20 degrees)

0 50 100 150

Pose (30 degrees)

−50 0 50 100 150

Pose (40 degrees)

−50 0 50 100 150

Pose (50 degrees)

−50 0 50 100 150

Pose (60 degrees)

−50 0 50 100 150

Pose (70 degrees)

−50 0 50 100 150

Pose (80 degrees)

Error Analysis

0 5 10 150

Noise Standard Deviation (px)

T. Error (0 degrees)

Interval (99%)

0 5 10 150

Noise Standard Deviation (px)E

Error Analysis

0 5 10 150

R. Error (0 degrees)

Interval (99%)

0 5 10 150

Noise Standard Deviation (px)R

Detection of Hand

Reliable and robust

Low error in pose estimation with little noise

Experimental dataset for adjusting theoreticalhand model

Kinematic Synthesis

Generalized inverse kinematics problem

Changing structural parameters to solveproblem

Motion-to-form problem

Very complex and generally unsolved problem

State of the Art

Simplifying the problemGroebner basesElimination theoryLinear product decomposition

State of the Art

Revolute joint structures solved

2R: 2 solutions (A. Perez andJ.M. McCarthy 2003)

State of the Art

2R: 2 solutions (A. Perez andJ.M. McCarthy 2003)3R: 13 solutions in an interval(Lee et al. 2004)

State of the Art

2R: 2 solutions (A. Perez andJ.M. McCarthy 2003)3R: 13 solutions in an interval(Lee et al. 2004)4R: Unsolved

State of the Art

2R: 2 solutions (A. Perez andJ.M. McCarthy 2003)3R: 13 solutions in an interval(Lee et al. 2004)4R: Unsolved...26R: Unsolved (our problem)

Kinematic Chain

Articulated system formed by jointsDifferent topologies

SerialParallelLoopTree (hand model)

Common Joints

Branch 2

Branch 1

Plucker Coordinates

Line representation: S = s︸︷︷︸

orientation

+ǫ s0︸︷︷︸

moment

Only 2 representations per line (s and −s)

Dual Quaternions

Homogeneous Matrix [H]︸︷︷︸

16 components

7−→ Dual Quaternion Q︸︷︷︸

8 components

Element of Clifford algebraUnit dual quaternions

Represents 3D displacements, points and linesTwo constraints: qq∗ = 1 and q · q0 = 0

Exponential Map

Based on Lie Algebra

Allows joints to be written as1-parameter screws

Joint parameters: θ = θ + ǫdPrismatic: θ = 0Revolute: d = 0

S(θ) = eθ

2S = cos

2+ sin

Forward Kinematics

Determines end effector position based onjoints

Direct calculation for serial kinematictopologies

T (θ) = S1(θ1)S2(θ2) · · · Sn(θn)G

= eθ12S1e

θ22S2 · · · e

= (cosθ1

2+ sin

2S1) · · · (cos

2+ sin

Forward Kinematics

∆θ = θ − θref + ǫ(d − dref )

Q(∆θ) = S1(∆θ1)S2(∆θ2) · · · Sn(∆θn)

= e∆θ1

∆θ22

S2 · · · e∆θn

= (cos∆θ1

2+ sin

∆θ1

· · · (cos∆θn

2+ sin

∆θn

Design Equations

Input: task defined by m − 1 relativetransformations P1j = TjT

−11 , j = 2, . . . , m

Variables: joint axes and joint anglesEquations:

S1j(∆θ1j)S2j(∆θ2j)S3j(∆θ3j)

Sk4j(∆θk

4j) · · · Sk7j(∆θk

7j) − Pk1j = 0, k ∈ 1, 2,

S1j(∆θ1j)S2j(∆θ2j)S3j(∆θ3j)

Sk4j(∆θk

4j) · · · Sk8j(∆θk

8j) − Pk1j = 0, k ∈ 3, 4, 5,

j = 2, ..., m

Non-linear Solver

System Dimension

Relations between variables and equationsdecrease dimension

Equations use 3 algebraic sets:

Set Comp Ind Notesθ 1 1 Periodic with period 2π

S = s + ǫs0 6 4 s · s0 = 0 , ‖s‖ = 1

Q = q + ǫq0 8 6 q · q0 = 0 , qq∗ = 1

System Dimension

System unknowns:

n0x = r( 4

︸︷︷︸

Structural

+(m − 1)︸︷︷︸

nx = r(6 + (m − 1))

System functions:

n0f = b6(m − 1)

nf = b8(m − 1)

System Dimension

System must be solvable: n0f ≥ n0

6b − r+ 1 > 0

6b − r > 0

Multiple solution possibilities

System Reduction

System can be reduced to subsystems that canbe solved

Allows reducing the dimension of the system

b r n0x = n0

f nx nf m Notes5 26 780 832 1092 27 Full hand model2 11 528 550 726 45 (4R, 4R) fingers1 5 120 130 170 21 5R, no common

Solution Bound

High dimensionality of the system

Important non-linearity of the equationsBezout’s theorem: D = d1d2 · · · dn

Serial chain: d = (3r)6 (A. Perez 2003)

di = 216(m−1) , i ∈ 1, 2,

di = 246(m−1) , i ∈ 3, 4, 5,

D =5∏

di ≈ 101058

Numerical Solver

Global optimizer

Allow overdetermined systems

Numerical Solver

Global optimizer

Allow overdetermined systemsLevenberg-Marquadt Algorithm

Local optimizerLeast squaresVery localVery difficult to parallelize

Numerical Solver

Global optimizer

Allow overdetermined systemsLevenberg-Marquadt Algorithm

Local optimizerLeast squaresVery localVery difficult to parallelize

Genetic Algorithm“Global” optimizer(meta-heuristics)Easy to implementVery hard to tuneExtremely parallel

Implementation

Hybrid solverReduces search space by minimizing withLevenberg-Marquadt algorithmGlobal optimizer by using genetic algorithm

Implementation

Results

Solver

Parallel algorithm

Bottleneck in QR decomposition fromLevenberg-Marquadt algorithm

Average run:

Generations Runtime (hours)38 60

Solver

Mutation

Solution

Solutions

Non-anthropomorphic structure

Moves finger positions exactly the same as ahandNeeds support for constraints to distinguish

More complicated solver

Solutions

Non-anthropomorphic structure

Moves finger positions exactly the same as ahandNeeds support for constraints to distinguish

More complicated solver

Solutions

Results

Softwarelibdq: Library for manipulating dual quaternionsKinematic hand model solverSoftware to track fingernail markers

PublicationE. Simo-Serra, F. Moreno-Noguer and A.Perez-Gracia, “Design of non-anthropomorphicrobotic hands for anthropomorphic tasks”,ASME-IDETC, accepted.

Conclusions

Major problems in engineeringPnP or pose estimation problemMotion-to-form or dimensional kinematic synthesisproblem

Results reflect maturity in both problems

Conclusions

Results reflect maturity in both problemsHand detection

Low computation costReliable with slow movements

Conclusions

Results reflect maturity in both problemsHand detection

Low computation costReliable with slow movements

Kinematic synthesisVery high complexityLarge number of solutionsNon-anthropomorphic solutions found

Conclusions

Objectives1 Defined theoretical hand model2 Detected real hand characteristics with computer

vision3 Adjusted theoretical model to experimental data

Conclusions

Objectives1 Defined theoretical hand model2 Detected real hand characteristics with computer

vision3 Adjusted theoretical model to experimental data

Anthropomorphic solution not foundFirst project to attempt dimensional kinematicsynthesis of tree-like topologiesKinematic synthesis is generally unsolved (3R isnot algebraically solved)Upper bound of 101058 solutions

Future Work

Flexible generic tree-like topology dimensionalkinematic synthesis solver

Exchangeable backendsScripting supportSupport for constraints

Future Work

Flexible generic tree-like topology dimensionalkinematic synthesis solver

Exchangeable backendsScripting supportSupport for constraints

Study of design equationsHow many solutions are there?What causes loss of dimensionality?How are solutions (and local minima) distributed?Can the a reduced model of the search space beused for initial estimation?Are matrix representation better at convergence?

Thanks

Questions?

Kinematic Model of the Hand using Computer VisionEnglish).pdf · Kinematic hand model solver...

Documents

Transcript of Kinematic Model of the Hand using Computer VisionEnglish).pdf · Kinematic hand model solver...

Kinematic and Motion

Mtls p kinematic

Kinematics – Kinematic Equations

Introduction and Kinematic Modeling - · PDF fileIntroduction and Kinematic Modeling ... car-like Robotics 1 2 . Wheeled mobile ... Kinematic model of WMR

Dc Motores - Kinematic

Animacion-dmu kinematic

Geometric Jacobian Analytical Jacobian Kinematic ... · Geometric Jacobian Analytical Jacobian Kinematic singularities Kinematic redundancy Inverse differential kinematics Inverse

1 04 - kinematic equations 04 - kinematic equations - large deformations and growth.

KINEMATIC AND PSEUDO-KINEMATIC GPS SURVEYING (TEST ...

Surgical treatment of acute ﬁngernail injuries · e-mail: pierluigi.tos@unito.it F. Catalano Department of Surgical Specialities, UOD Plastic and Reconstructive Surgery, University

Kinematic Concepts

Kinematic cellularautomata

TC Kinematic

Inverse Kinematic Solvers

Dr Miquel Noguer i Alonso and Sonam Srivastava Wright ...

kinematic inversions

Kinematic Robot

kinematic synthesis

Kinematic KKiinneemmaattiicc Kinematic Modeling ...

2.4 Kinematic Equations