Tracking and Modeling Non-Rigid Objects with Rank Constraints
Enforcing Constraints for Human Body Tracking
description
Transcript of Enforcing Constraints for Human Body Tracking
![Page 1: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/1.jpg)
Enforcing Constraints for Human Body Tracking
David DemirdjianArtificial Intelligence Laboratory, MIT
![Page 2: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/2.jpg)
WOMOT 2003
Goal Real-time articulated body tracking from
stereo accounting for constraints on pose
![Page 3: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/3.jpg)
WOMOT 2003
Approach
Differential tracking: assuming the articulated body pose t-1 is known, estimate the pose t (or
equivalently the set of limb rigid motions i=(tii) between poses t-1 and t) that minimizes the distance between the articulated model and the observed 3D data
tracking as a constrained optimization problem
![Page 4: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/4.jpg)
WOMOT 2003
Approach Differential tracking: assuming the articulated
body pose t-1 is known, estimate the pose t (or equivalently the set of limb rigid motions i=(tii) between poses t-1 and t) that minimizes the distance between the articulated model and the observed 3D data
tracking as a constrained optimization problem– Solve unconstrained optimization problem– Project solution on constraint surface
![Page 5: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/5.jpg)
WOMOT 2003
Projection-based approach
unconstrained optimum)
human motion manifold
![Page 6: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/6.jpg)
WOMOT 2003
Approach
Estimate limb motions i=(tii) independently using standard multi-object tracking algorithm
Projection: find the closest body motion =(i’) to =(i) that satisfies human body constraints: – articulated constraints – other constraints: joint limit, …
![Page 7: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/7.jpg)
WOMOT 2003
Previous work
Particle sampling: Sidenbladh & al. ECCV’00 Sminchisescu & Triggs CVPR’01
Differential tracking: Plankers & Fua ICCV’99Jojic & al. ICCV’99Delamarre & Faugeras ICCV’99
![Page 8: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/8.jpg)
WOMOT 2003
Plan
Unconstrained problem Articulated constraints enforcing Other constraints Tracking results Application (Multimodal interface) Conclusion
![Page 9: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/9.jpg)
WOMOT 2003
Multi-object tracking Assuming the articulated body pose t-1 is known, estimate
the set of limb rigid motions i=(tii) minimizes the distance between the (moved) limb and the observed 3D data
Consists in estimating limb motions i=(tii) independently:– Estimate visible 3D mesh of each limb– Current implementation uses the ICP algorithm to
register each limb w.r.t 3D data
![Page 10: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/10.jpg)
WOMOT 2003
Iterative Closest Point
3D registration– find the rigid transformation that maps shape St (limb model) to
shape Sr (3D data)
SrSt
),(minarg1
2*
n
irt SSGd
![Page 11: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/11.jpg)
WOMOT 2003
Iterative Closest Point
Matching points• For all points in St, we search for the closest point in Sr by
computing the distance and keep the closest
SrSt
![Page 12: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/12.jpg)
WOMOT 2003
Iterative Closest Point
Energy function minimization• Estimate the rigid transformation that minimizes the sum of
squared distances between corresponding matched points
SrSt
![Page 13: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/13.jpg)
WOMOT 2003
Iterative Closest Point
Energy function minimization• Estimate the rigid transformation that minimizes the sum of
squared distances between corresponding matched points
SrSt
![Page 14: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/14.jpg)
WOMOT 2003
Iterative Closest Point
Optimal rigid transformation (and uncertainty ) found by combining all the elementary displacements
![Page 15: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/15.jpg)
WOMOT 2003
Plan
Unconstrained problem Articulated constraints enforcing Other constraints Tracking results Application (Multimodal interface) Conclusion
![Page 16: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/16.jpg)
WOMOT 2003
ProjectionThe unconstrained optimal body motion is given by =(1, 2 … N)With uncertainty =(1, 2 … N)
)()( 12 TE
with =(1’, 2’ … N’) satisfying articulated constraints
Articulated constraints enforcement: find that minimizes the Mahalanobis distance:
![Page 17: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/17.jpg)
WOMOT 2003
Articulated motion estimation
If Mij is a joint between objects i and j:
Mij joint
(Ri,ti)
(Rj,tj)obj. i
obj. j
Motion of point Mij
on limb iMotion of point Mij
on limb j=
)(')(' ijjiji MM
![Page 18: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/18.jpg)
WOMOT 2003
Articulated motion estimation
If Mij is a joint between objects i and j:
Mij joint
(Ri,ti)
(Rj,tj)obj. i
obj. j
Motion of point Mij
on limb iMotion of point Mij
on limb j=
'''' jijjiiji tMRtMR
)(')(' ijjiji MM
![Page 19: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/19.jpg)
WOMOT 2003
Articulated motion estimation
If Mij is a joint between objects i and j:
Mij joint
(Ri,ti)
(Rj,tj)obj. i
obj. j
Motion of point Mij
on limb iMotion of point Mij
on limb j=
0'')''(][
0'']''[
')]'[(')]'[(
''''
jijiij
jiijji
jijjiiji
jijjiiji
ttM
ttM
tMItMI
tMRtMR
[.]x denotes skew-symmetric matrix
)(')(' ijjiji MM
![Page 20: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/20.jpg)
WOMOT 2003
Articulated motion estimation
If Mij is a joint between objects i and j:
Mij joint
(Ri,ti)
(Rj,tj)obj. i
obj. j
Motion of point Mij
on limb iMotion of point Mij
on limb j=
0'')''(][ jijiij ttM
[.]x denotes skew-symmetric matrix
)(')(' ijjiji MM
![Page 21: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/21.jpg)
WOMOT 2003
Articulated motion estimation
If Mij is a joint between objects i and j:
Mij joint
(Ri,ti)
(Rj,tj)obj. i
obj. j
Motion of point Mij
on limb iMotion of point Mij
on limb j=
0 ijS
)(')(' ijjiji MM
0'')''(][ jijiij ttM
=(1’, 2’ … N’)
![Page 22: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/22.jpg)
WOMOT 2003
Articulated motion estimation
If Mij is a joint between objects i and j:
Mij joint
(Ri,ti)
(Rj,tj)obj. i
obj. j
Motion of point Mij
on limb iMotion of point Mij
on limb j=
(Stack for all joints)0
)(')(' ijjiji MM
0 ijS
0'')''(][ jijiij ttM
=(1’, 2’ … N’)
![Page 23: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/23.jpg)
WOMOT 2003
Articulated motion estimation
All the joint constraints can be written as a linear constraint:
0
is a linear combination of vectors in the nullspace of Therefore there exists a matrix V such that:
V
V can be estimated by SVD of
![Page 24: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/24.jpg)
WOMOT 2003
Articulated motion estimation
)()( 12 VVE T
111 )( TT VVVVP
)()( 12 TE
unconstrainedmotion
articulatedmotion
Find minimum of E2 in nullspace of
P
…
(linear projection)
![Page 25: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/25.jpg)
WOMOT 2003
Plan
Unconstrained problem Articulated constraints enforcing Other constraints Tracking results Application (Multimodal interface) Conclusion
![Page 26: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/26.jpg)
WOMOT 2003
Other constraints
Constraints:– Static: Joint angle bounds, gravity law, …– Dynamic: Maximum velocity, …
Motivation:– Using more constraints to reduce local minima
and therefore increase tracking robustness– Application context can reduce tremendously
the dimension of the pose space
![Page 27: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/27.jpg)
WOMOT 2003
Other constraints
)()( *1*2 TE
Pose constraints modeled by a (user-defined) function f, such that valid poses correspond to f()>0
ex: f()=min(g1(), g2(), … gN()) with g1() = angle(l-arm, l-forearm) – min_angle
g2() = max_angle - angle(l-arm, l-forearm)….
Constraints enforcement: find * that minimizes the Mahalanobis distance:
with * satisfying Ft-1(*)=f( *(t-1))>0
![Page 28: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/28.jpg)
WOMOT 2003
Other constraints
)()( *1*2 TE
articulated motionarticulated constrainedmotion
** V
)()(
)()(*1*2
*1*2
VVE
VVVVETT
T
with (local parameterization)
![Page 29: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/29.jpg)
WOMOT 2003
Constrained optimization algorithmAlternate between binary and stochastic searches
![Page 30: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/30.jpg)
WOMOT 2003
Constrained optimization algorithmAlternate between binary and stochastic searches
![Page 31: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/31.jpg)
WOMOT 2003
Constrained optimization algorithmAlternate between binary and stochastic searches
E2 = E0
![Page 32: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/32.jpg)
WOMOT 2003
Constrained optimization algorithmAlternate between binary and stochastic searches
E2 = E1
![Page 33: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/33.jpg)
WOMOT 2003
Constrained optimization algorithmAlternate between binary and stochastic searches
![Page 34: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/34.jpg)
WOMOT 2003
TRACKING SEQUENCE
![Page 35: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/35.jpg)
WOMOT 2003
Future work
Quantitative measurement(comparing results with tethered motion capture system)
Appearance/Shape information(learning color distribution + shape of limbs)
Motion/gesture(including dynamic constraints)
Learning human motion constraints (instead of giving them explicitly.. [ICCV’03])
![Page 36: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/36.jpg)
WOMOT 2003
Applications Multimodal Human-Computer Interaction
(gesture + speech)
![Page 37: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/37.jpg)
WOMOT 2003
![Page 38: Enforcing Constraints for Human Body Tracking](https://reader035.fdocuments.in/reader035/viewer/2022062521/568167f4550346895ddd6b58/html5/thumbnails/38.jpg)
WOMOT 2003
Conclusion
Projection-based approach for articulated body tracking– articulated constraints enforced by (linearly)
projecting unconstrained limb motion on articulated motion manifold
– other constraints enforced using a stochastic constrained optimization algorithm