Non-Rigid Articulated Point Set Registration With Local ... · Tr( YTMY)+2Tr(WTGMY)+Tr(WTGMGW), (7)...

8
Non-rigid Articulated Point Set Registration with Local Structure Preservation Song Ge and Guoliang Fan School of Electrical and Computer Engineering Oklahoma State University, Stillwater, OK 74078 {song.ge,guoliang.fan}@okstate.edu Abstract We propose a new Gaussian mixture model (GMM)- based probabilistic point set registration method, called Lo- cal Structure Preservation (LSP), which is aimed at com- plex non-rigid and articulated deformations. LSP integrates two complementary shape descriptors to preserve the lo- cal structure. The first one is the Local Linear Embedding (LLE)-based topology constraint to retain the local neigh- borhood relationship, and the other is the Laplacian Coor- dinate (LC)-based energy to encode the local neighborhood scale. The registration is formulated as a density estima- tion problem where the LLE and LC terms are embedded in the GMM-based Coherent Point Drift (CPD) framework. A closed form solution is solved by an Expectation Maximiza- tion (EM) algorithm where the two local terms are jointly optimized along with the CPD coherence constraint. The experimental results on a challenging 3D human dataset show the accuracy and efficiency of our proposed approach to handle non-rigid highly articulated deformations. 1. Introduction Point set registration is a fundamental issue in computer vision. The registration techniques usually fall into two categories: rigid and non-rigid depending on the under- lying transformation model. Iterative Closest Point (ICP) [2, 26] is a classic rigid registration method which itera- tively assigns correspondence and then finds the least square transformation by using the estimated correspondence. For non-rigid registration, shape features are commonly used for correspondence initialization [27, 11, 12] or directly in- volved in the matching process [22, 20]. Gaussian Mixture Model (GMM)-based registration algorithms are becoming an important category, where the point sets are represented by density functions and then registration is cast as either a density estimation problem [4, 5, 16, 15] or an align- This work is supported by the Oklahoma Center for the Advancement of Science and Technology (OCAST) under grant HR12-30 and the Na- tional Science Foundation (NSF) under grant NRI-1427345. ment between two distributions [8, 9]. Coherent Point Drift (CPD) [16, 15] is an effective GMM-based non-rigid reg- istration algorithm that imposes a coherence constraint to preserve the topology of the point sets. On the other hand, articulated point set registration is be- coming an attractive topic, e.g, human pose estimation [23] or body shape modeling [21]. Some initial conditions or new constraints are needed to cope with the complicated non-rigid articulated deformation. In [23], a pose initial- ization from a large training dataset is used to reduce the articulated deformation between two point sets to be regis- tered. In [13], Laplacian eigenfunctions are used to align two voxels represented by spectral graphs for registration initialization. By incorporating the global (CPD) and lo- cal topological constraints (LLE) into a GMM-based frame- work, the Global-Local Topology Preservation (GLTP) al- gorithm was proposed in [6] which obtains promising per- formance. Some approaches treat the articulated deforma- tion as a chain of constrained rigid motions, where pre- segmentation and locally rigid assumption are usually re- quired. An articulated ICP (AICP) algorithm was proposed in [18] which adopts a divide-and-conquer strategy to iter- atively estimate the articulated structure by assuming it is partially rigid. The similar strategy was applied in [7] by using a GMM-based rigid registration algorithm instead of ICP. By using the exponential maps based parametrization, the articulated deformation is effectively embedded in the GMM-based framework in [24]. We propose a new GMM-based point set registration al- gorithm, called Local Structure Preservation (LSP), to deal with non-rigid highly articulated deformations, such as 3D human data. Without involving the locally rigid assump- tion or requiring any initial conditions, LSP is non-rigid both globally and locally which is more flexible and gen- eral to handle the non-rigid articulated deformations than those with the locally rigid assumption (e.g. [18, 7, 24]). The key is to involve two local shape descriptors, the Local Linear Embedding (LE) and Laplacian Coordinate (LC), to preserve the local structure in terms of the neighborhood relationship and the neighborhood scale, respectively. The

Transcript of Non-Rigid Articulated Point Set Registration With Local ... · Tr( YTMY)+2Tr(WTGMY)+Tr(WTGMGW), (7)...

  • Non-rigid Articulated Point Set Registration with Local Structure Preservation

    Song Ge and Guoliang Fan∗

    School of Electrical and Computer EngineeringOklahoma State University, Stillwater, OK 74078

    {song.ge,guoliang.fan}@okstate.edu

    Abstract

    We propose a new Gaussian mixture model (GMM)-based probabilistic point set registration method, called Lo-cal Structure Preservation (LSP), which is aimed at com-plex non-rigid and articulated deformations. LSP integratestwo complementary shape descriptors to preserve the lo-cal structure. The first one is the Local Linear Embedding(LLE)-based topology constraint to retain the local neigh-borhood relationship, and the other is the Laplacian Coor-dinate (LC)-based energy to encode the local neighborhoodscale. The registration is formulated as a density estima-tion problem where the LLE and LC terms are embedded inthe GMM-based Coherent Point Drift (CPD) framework. Aclosed form solution is solved by an Expectation Maximiza-tion (EM) algorithm where the two local terms are jointlyoptimized along with the CPD coherence constraint. Theexperimental results on a challenging 3D human datasetshow the accuracy and efficiency of our proposed approachto handle non-rigid highly articulated deformations.

    1. Introduction

    Point set registration is a fundamental issue in computervision. The registration techniques usually fall into twocategories: rigid and non-rigid depending on the under-lying transformation model. Iterative Closest Point (ICP)[2, 26] is a classic rigid registration method which itera-tively assigns correspondence and then finds the least squaretransformation by using the estimated correspondence. Fornon-rigid registration, shape features are commonly usedfor correspondence initialization [27, 11, 12] or directly in-volved in the matching process [22, 20]. Gaussian MixtureModel (GMM)-based registration algorithms are becomingan important category, where the point sets are representedby density functions and then registration is cast as eithera density estimation problem [4, 5, 16, 15] or an align-

    ∗This work is supported by the Oklahoma Center for the Advancementof Science and Technology (OCAST) under grant HR12-30 and the Na-tional Science Foundation (NSF) under grant NRI-1427345.

    ment between two distributions [8, 9]. Coherent Point Drift(CPD) [16, 15] is an effective GMM-based non-rigid reg-istration algorithm that imposes a coherence constraint topreserve the topology of the point sets.

    On the other hand, articulated point set registration is be-coming an attractive topic, e.g, human pose estimation [23]or body shape modeling [21]. Some initial conditions ornew constraints are needed to cope with the complicatednon-rigid articulated deformation. In [23], a pose initial-ization from a large training dataset is used to reduce thearticulated deformation between two point sets to be regis-tered. In [13], Laplacian eigenfunctions are used to aligntwo voxels represented by spectral graphs for registrationinitialization. By incorporating the global (CPD) and lo-cal topological constraints (LLE) into a GMM-based frame-work, the Global-Local Topology Preservation (GLTP) al-gorithm was proposed in [6] which obtains promising per-formance. Some approaches treat the articulated deforma-tion as a chain of constrained rigid motions, where pre-segmentation and locally rigid assumption are usually re-quired. An articulated ICP (AICP) algorithm was proposedin [18] which adopts a divide-and-conquer strategy to iter-atively estimate the articulated structure by assuming it ispartially rigid. The similar strategy was applied in [7] byusing a GMM-based rigid registration algorithm instead ofICP. By using the exponential maps based parametrization,the articulated deformation is effectively embedded in theGMM-based framework in [24].

    We propose a new GMM-based point set registration al-gorithm, called Local Structure Preservation (LSP), to dealwith non-rigid highly articulated deformations, such as 3Dhuman data. Without involving the locally rigid assump-tion or requiring any initial conditions, LSP is non-rigidboth globally and locally which is more flexible and gen-eral to handle the non-rigid articulated deformations thanthose with the locally rigid assumption (e.g. [18, 7, 24]).The key is to involve two local shape descriptors, the LocalLinear Embedding (LE) and Laplacian Coordinate (LC), topreserve the local structure in terms of the neighborhoodrelationship and the neighborhood scale, respectively. The

  • two local regularization terms are unified with the CPD co-herence constraint into the GMM-based density estimationframework. A joint optimization of three terms is achievedby an Expectation and Maximization (EM) algorithm forMaximum Likelihood (ML) optimization.

    2. Proposed Method

    We first review the GMM-based registration frame-work along some existing regularization terms for non-rigidtransformation estimation. Then we will introduce a newlocal regularization term ELSP . Afterwards, we presentthe formulation and optimization of the proposed LSP al-gorithm followed by a discussion of parameter selection.

    2.1. Registration as Mixture Density Estimation

    We consider point set registration as a probability den-sity estimation problem, where one point set (Y =[y1, · · · ,yM ]T , ym ∈ RD) is treated as the template witha sparse distribution which presents the centroids from theGaussian mixture model (GMM), while the other point set(X = [x1, · · · ,xN ]T , xn ∈ RD) is served as the tar-get with a dense distribution, and M,N,D are the num-ber of points and the dimension of each point respectively.Then the goal is to find the optimal GMM parameters (e.g.the centroids which are controlled by unknown transforma-tion) to maximize the GMM posterior probability. We de-note the spatial transformation as T (Y,Θ), which can beconsidered as a function of Y with parameters Θ. To ac-count for outliers in X, a uniform component with weightω (0 ≤ ω ≤ 1) is added [15, 4]. For simplicity, we considerall Gaussian components are independently distributed withan equal isotropic variance σ2 and an equal weight, then thejoint GMM probability density function is written as

    p(X) =N∏

    n=1

    p(xn) =N∏

    n=1

    M+1∑

    m=1

    πmp(xn|m), (1)

    where p(x|m) = 1(2πσ2)

    D2exp(− ‖x−T (ym,Θ)‖

    2

    2σ2 ) (m =

    1, · · · ,M ), p(x|m) = 1N for m = M + 1, πm =1−ωM

    (m = 1, · · · ,M ), and πM+1 = ω. Then registration is con-verted to the problem of finding Θ and σ2 that minimizesthe negative log-likelihood of (1).

    Following the EM algorithm for GMM-based clustering[3, 15], we can find the objective (E-step) as

    Q(Θ, σ2) =

    N∑

    n=1

    M∑

    m=1

    pold(m|xn)‖ xn − T (ym,Θ) ‖2

    2σ2

    +NpD

    2ln(σ2), (2)

    whereNp =∑N

    n=1

    ∑Mm=1 p

    old(m|xn) and pold(m|xn) arethe posterior probabilities that can be computed using the

    old GMM parameters based on the Bayes rule as

    pold(m|xn) =exp(− 1

    2‖ xn−T (ym,Θ)

    σold‖2)

    ∑Mi=1 exp(− 12‖

    xn−T (yi,Θ)σold

    ‖2) + (2πσ2)D2 ωM

    (1−ω)N

    .

    (3)Then in the M-step the new GMM parameters are updatedby minimizing (2). The EM algorithm performs iterativelyby alternating between E-step and M-step until it converges.The different type of transformations can lead to differentoptimization strategies.

    2.2. Non-rigid Transformation Regularization

    Particularly, Coherent Point Drift (CPD) [15] is a pow-erful and noteworthy GMM-based non-rigid registrationmethod where the underlying non-rigid transformationT (Y,Θ) is defined as the initial position Y plus a dis-placement function f(Y). The displacement function f ismodeled in a Reproducing Kernel Hilbert Space (RKHS)and the spatial smoothness regularization is defined as theFourier domain norm of f . It has been proved that the opti-mal function f which minimizes the objective (2) under thespatial smoothness constraint is given by a linear combina-tion of Gaussian kernel functions as

    T (Y,W) = Y +GW, (4)

    where GM×M is the Gaussian kernel matrix with elementgij = exp(− 12 ‖

    yi−yjβ ‖2) and WM×D is the weight ma-

    trix of the Gaussian kernel. The corresponding constraintwhich regularizes the weight matrix W has the form as

    EMC(W) = Tr(WTGW). (5)

    It was shown in [15] that the selection of the Gaussian ker-nel makes the regularization equivalent to the one in the Mo-tion Coherence Theory [25] which forces points to move to-gether as a group to keep the motion coherence. The motioncoherence is helpful to keep the overall spatial connectivityof a complicated point set during the registration. However,it may not handle the non-rigid highly articulated deforma-tion where the motion coherence assumption may be vio-lated.

    Recently some specific regularization terms were pro-posed and embedded in the GMM-based CPD frameworkfor improving registration performance. In [17], a Graph-Laplacian (GL) based regularization term, which enforcestransformation smoothness to avoid mis-matches duringregistration, is added to restrict a large displacement be-tween two neighboring points. This regularization termenhances the registration robustness under noise, outliersand occlusions. It mainly penalizes large displacements be-tween the adjacent neighbors, but does not consider the spa-tial relationship between the edges (constructed by neigh-boring points) which may not be effective to preserve the

  • local shape structure. On the other hand, the LLE-basedtopology constraint proposed in [6] is intended to preservethe neighborhood structure by retaining the local neighbor-hood relationship. This was found helpful to cope with non-rigid articulated deformations. However, the LLE-basedneighborhood structure is insensitive to the scaling whichmeans the scale of the neighborhood structure may not bepreserved well, leading to inaccurate correspondence esti-mation at body joints, where two adjacent segments are con-nected. However, joint estimation is critical for articulatedpose estimation. This motivates us to study more effectiveregularization for local structure preservation.

    2.3. Local Structure Preservation

    In this work, we try to preserve the local shape structurein terms of neighborhood structure along with the motioncoherence with respect to two aspects: spatial relative loca-tion and the size of the neighborhood structure. Our pro-posed LSP regularization term ELSP is built for preservingtwo local shape descriptors. One is the LLE-based descrip-tor in terms of preserving the spatial relationship among theneighboring points. The other one is a Laplacian coordinate(LC)-based descriptor to preserve the scale of the neigh-borhood structure which may not be encoded by the LLE-based descriptor. By preserving two local shape descriptors,ELSP has the form as

    ELSP = ELL + ELC , (6)

    where ELL and ELC denote the LLE-based and LC-basedregularization terms for local structure preservation respec-tively.

    For ELL we adopt the similar idea of GLTP [6]. By find-ing the K1 nearest neighbors of each point in Y accordingthe Euclidean distance and representing each point in Y bya weighted linear combination of its neighbors, the LLE-based regularization term in matrix form is

    ELL(W) =

    Tr(YTMY) + 2Tr(WTGMY) + Tr(WTGMGW), (7)

    where the M×K matrix C contains the neighborhood in-formation for each point in Y, the M×M matrix M =(I− Ĉ)(I− Ĉ)T , Ĉ is an M×M matrix with Ĉij = Cij ,for j ≤ K and Ĉij = 0 for j > K , I denotes the identitymatrix. And the objective is to find the optimal coefficientmatrix W which controls the non-rigid transformation tominimize (7).

    The Laplacian coordinate is a simple form of the differ-ential coordinate which has been used to describe the localshape structure in mesh deformation [19, 10]. Given a meshmodel (V,E), where V denotes the set of vertices and Edenotes the edge set, the Laplacian coordinate could be ex-

    pressed as

    L(vi) =∑

    (i,j)∈EAij(vi − vj), (8)

    where Aij is the weight coefficient and (i, j) denotes theedge constructed by vertices vi and vj . It is easy to showthat the Laplacian coordinate is sensitive with scaling whichis helpful to preserve the size of neighborhood structure.There are two steps to model the LC-based regularizationin terms of point set registration. First, find the K2 near-est neighbors of each point in Y according the Euclideandistance which is used to model the graph. Then the regu-larization term which measures the difference of the Lapla-cian coordinate after the non-rigid transformation has theform as

    ELC(W) =

    M∑m=1

    ‖L(ym)− L(ym +G(m, ·)W)‖2, (9)

    where G(m, ·) denotes the mth row of G. We can rewriteit in the matrix form as

    ELC(W) = Tr(WTGLTLGW), (10)

    where the M×M Laplacian matrix L = D − A, matrixA is the adjacency matrix of the graph, D is the degreematrix which is a diagonal matrix and its entries are columnsums of A [14]. For simplicity we set Aij = 1 if pointsyi and yj are neighbors and Aij = 0 otherwise. Now theobjective is to find the optimal weight matrix W to preservethe neighborhood structure by minimizing (10). We need topoint out the Laplacian coordinate is not rotation-invariant,therefore this strong constraint may be counter-productivefor the articulated deformation. However a balanced controlof ELL and ELC under CPD-based regularization makesthem complement each other to preserve the local structure.

    2.4. Algorithm and Optimization

    In this work we treat point set registration problem asa GMM density estimation problem. By incorporating themotion coherence and the local structure preservation regu-larization into general formulation (2), the registration ob-jective function can be written in the form as

    Q(W, σ2) =

    M,N∑

    m,n=1

    pold(m|xn)‖ xn − (ym +G(m, ·)W) ‖2

    2σ2

    +NpD

    2ln(σ2) +

    α

    2EMC(W) +

    λ

    2ELL(W) +

    γ

    2ELC(W),

    (11)

    where α, λ and γ are the trade-off parameters among theGMM matching term and the regularization terms. Thenwe can rewrite the objective function (11) in a matrix form.In order to obtain the weight matrix W which minimizes

  • the objective function, we take the derivative of the matrixform of (11) respect to W and set it equal to zero, and thenwe obtain

    ∂Q

    ∂W= −GPX +Gd(P1)Y +Gd(P1)GW + σ2αGW

    + σ2λGMY + σ2λGMGW + σ2γGLTLGW = 0, (12)

    where the M×N matrix P has the element pold(m|xn),d(v) denotes the diagonal matrix formed from the vectorv and 1 denotes the column vector of all ones. Then W canbe obtained by solving a linear system:

    [d(P1)G + σ2αI+ σ2λMG+ σ2γLTLG]W

    = PX− (d(P1) + σ2λM)Y. (13)

    Similarly σ2 can be obtained by taking the correspondingderivative of the matrix form of (11) and setting it equal tozero. We have

    σ2 =1

    NpD(Tr(XTd(PT1)X)− 2tr(YTPX)

    − 2Tr(WTGTPX) + Tr(YTd(P1)Y)+ 2Tr(WTGT d(P1)Y) + tr(WTGT d(P1)GW)). (14)

    After the EM algorithm converges, the transformed pointset is Ytran = Y+GW and the probability of correspon-dence is stored in matrix P. The EM optimization processis similar with that in [15, 6].

    2.5. Parameter Discussion

    The proposed algorithm contains several free parame-ters: ω, K1, K2, β, α, λ and γ. ω (0 ≤ ω ≤ 1) reflects theproportion of the points in X which are treated as outliers ornoise, and β is the width of the Gaussian kernel. K1 is thenumber of neighbors used to compute the LLE coefficientmatrix C and K2 is the number of neighbors to compute theLaplacian matrix L. The values of K1 and K2 are relatedto the density and distribution of the points in the templateY. Normally, the denser Y is, the larger values should beor vice versa. α, λ and γ are the three trade-off parametersamong the motion coherence and the local structure preser-vation terms, which also control the balance between theregularization terms and the GMM-based matching term.

    In the proposed LSP algorithm, the three regularizationterms are incorporated together to play a mutually helpfulrole to preserve the local structure along with the spatialcoherence, which is important to obtain the accurate corre-spondence estimation for complicated non-rigid articulateddeformations. During the EM optimization process, the cor-respondence estimation during the initial iterations is crit-ical to guide the template point set to move to the targetalong the desirable directions. Therefore, we set a relativelarge value for λ comparing with α and γ in the initial stageto strengthen the local structure preservation under motion

    coherence. However the strong regularization may discour-age the registration to conform to local details. So duringthe latter iterations, we slowly reduce α, λ and γ to allowthe points moving close to the corresponding points. In par-ticular, for a clean target point set, we finally decrease theα and γ down to zero and only preserve the LLE-based lo-cal structure to provide good registration results with localdetail shape representation.

    3. Experiments

    We have implemented the proposed algorithm in Mat-lab and tested it on a 3D human mesh dataset which cre-ates a challenging non-rigid articulated registration prob-lem where highly articulated poses are involved. The targetpoint sets (12500 points for each target) are extracted fromthe SCAPE human mesh dataset [1] which includes 35 dif-ferent highly articulated poses and are fully registered. Thetemplate point set we use is a T-pose model (2150 points)provided by [24] which has significant shape differencesand articulated deformations compared with the target pointsets.

    3.1. Data Preparation

    Since the template and target models are captured fromdifferent subjects and also have different number of points,it is difficult to obtain the ground-truth correspondences.Thus a quantitative result in terms of registration error isnot available in this experiment. Instead we use the accu-racy of body segment labeling to evaluate the registrationperformance. To do so, we need to create the ground-truthsegment labels for the template model and all target models.The datasets provide the ground-truth joint positions bothfor the “T-pose” template model and one target model, fromwhich we can create the segment labels. The general T-posetemplate model with joint positions and segment labels isshown in Fig. 1 and the target model with joint positionsand segment labels, called “initial target pose” is shown inFig. 2(a)(b).

    Figure 1. The T-pose template model with given joint positions(left) and segment labels (right).

  • Since the SCAPE mesh data are fully aligned and reg-istered, the body segment labels can be easily transferredfrom one labeled target model to all others. One exampleis shown in Fig. 2 where the segment labeling informationfrom one labeled target model is transferred to another tar-get model of an arbitrary pose, as shown Fig. 2(c)(d). Withthe creation of ground-truth segment labels for all targetmodels, we can evaluate the registration performance quan-titatively by calculating the labeling accuracy of each bodysegment after registration. For each point in the templatemodel, we propagate its segment label to the correspondingpoint in the target model by estimated correspondence. Ifthis assigned segment label is as the same as the ground-truth label, we treat it as the correct segment label as shownin Fig. 3. Then the labeling accuracy is calculated as thepercentage of the points with correct segment labels overall labeled points. It is worth mentioning that the labelinginformation is not involved during the registration processand is only for performance evaluation.

    (a) (b) (c) (d)

    Figure 2. The creation of ground-truth segment labels. (a) The“initial pose” target model with given joints positions; (b) The la-beled target model from (a); (c) A target model of an arbitrarypose; (d) The labeled target model obtained by label transferringfrom (b).

    Incorrect labels

    Correct labels

    The labeled target model after registration

    The target model with ground-truth labels

    Figure 3. The illustration of the calculation of labeling accuracyafter registration where all labeled points in the registered targetmodel (left) are checked with respect to the ground truth labelscreated for the target model according to Fig. 2 (right).

    3.2. Results on 3D Human Data

    In order to evaluate the performance of our proposed al-gorithm, we performed a quantitative comparison againstseveral recent algorithms which are able to handle 3D data.To the best of our knowledge, there is little study and nu-merical results on 3D non-rigid point set registration, es-pecially for those with complicated articulated deforma-tions. Therefore, we mainly compared our proposed algo-rithm LSP with CPD [15], GLTP [6], and the algorithm pro-posed in [17] (for simplicity we refer it as GL-CPD and forfair comparison we implemented it without feature-basedcorrespondence initialization). In the experiments, we setα = 40, λ = 5 × 106, β =

    √2, γ = 10, ω = 0 and

    K1 = K2 = 15. The extra small or large values of K1 andK2 cannot describe a good local neighborhood structure. Inthe experiment, we found the optimal values of K1 and K2are between 10 to 20 and the registration results are not verysensitive with the values in that range.

    0.5

    0.6

    0.7

    0.8

    0.9

    1

    Head Torso Arms Legs Total

    CPD GLTP GL-CPD LSP

    Figure 5. Quantitative comparison of CPD, GLTP, GL-CPD andLSP for the SCAPE data in terms of the accuracy of segment la-beling at some body segments and the whole body.

    We first present the qualitative comparison in Fig. 4. It isshown that when articulated deformation is not significantbetween the template and the target, such as the first andsecond poses, all four algorithms (CPD, GLTP, GL-CPDand LSP) can obtain the similar results of correspondenceestimation (segment labels shown in columns (b-e)). How-ever, in the cases of highly articulated deformations, e.g.,poses 3 to 6, significant correspondence errors are observedfrom the CPD and GL-CPD results around the head, limbsand body joints, leading to large registration errors in thoseareas. GL-CPD is helpful to improve the results comparedwith CPD, but it is not sufficient enough to keep the lo-cal structure in some cases (e.g. pose 4 to 6). GLTP pro-vides better correspondence estimation across all poses atthe global scale. But at the local scale, the correspondenceestimation at body joints, the connection part between ad-jacent segments, is not very accurate due to the scaling in-

  • Figure 4. Qualitative comparison of CPD, GLTP, GL-CPD and LSP for the SCAPE data. Column (a) shows the six target models withdifferent poses (from the first to the sixth rows); columns (b-e) show the correspondence estimation results of CPD, GLTP, GL-CPD andLSP respectively (the color labels are transferred from the template model through the estimated correspondence); columns (f-i) show theregistration results of CPD, GLTP, GL-CPD, and LSP respectively.

  • (a) (b) (c)

    Figure 6. Detailed comparison of the segment labeling of GLTP(column (a)), GL-CPD (column (b)) and LSP (column (c)) at thearms, torso and legs (the circled areas).

    sensitivity of the LLE-based local shape descriptor. On theother hand, LSP provides stable and accurate correspon-dence estimation across all poses which directly contributesto good registration results around the whole body and atlocal segments. The improvements in the joint areas are ob-vious, compared with other three algorithms.

    As an alternative of correspondence evaluation, the meanlabeling accuracy of body segments over all 35 target mod-els is compared among CPD, GLTP, GL-CPD and LSP inFig. 5. From Fig. 5 we can see that GLTP, GL-CPD andLSP are significantly better than CPD, which shows the ad-vantages by incorporating additional regularization termsfor dealing with non-rigid articulated registrations. For thearms and torso, GLTP achieves better results than GL-CPD,and both GLTP and GL-CPD have similar results on theleg segments. The main reason that GLTP outperforms GL-CPD at the arms and torso is due to the fact that large ar-ticulated deformations may not be represented well by thegraph-based distance constraint in GL-CPD. On the otherhand, GL-CPD better preserves the segment length than

    GLTP, especially near the knees on the two legs, leadinga better labeling accuracy at the legs. LSP outperformsthe other three algorithms at all segments. The quantita-tive results illustrate the advantage of the joint use of twolocal shape descriptors to handle the complicated non-rigidarticulated deformation. More detailed comparison resultsbetween GLTP, GL-CPD and LSP in terms of correspon-dences estimation at local segments are shown in Fig. 6.

    4. Conclusion

    We have presented a new GMM-based point set registra-tion algorithm, referred to as Local Structure Preservation(LSP), which is targeted on complex non-rigid and articu-lated deformations. The proposed LSP algorithm embracesthe original CPD constraint and more importantly, intro-duces dual local regularization terms to preserve the localstructure for non-rigid registration. One is the LLE-basedshape descriptor and the other is the LC-based neighbor-hood structure. The joint use of them in the CPD frameworkdemonstrates better flexibility and capability of coping withnon-rigid and articulated deformations while retaining thesize of each segment and spatial conference. Comparedwith the several related algorithms, including CPD, GLTPand GL-CPD, LSP is able to provide more accurate regis-tration results around joints where two adjacent segmentsare connected. This is especially important in the case ofarticulated pose estimation. The experiment results on a setof 3D human data manifest the advantages of our LSP algo-rithm for handling non-rigid and highly articulated body de-formations. Our future research will be extended to depth-based articulated pose estimation as well as the registrationof non-rigid and articulated 2D/3D objects.

    References

    [1] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun,J. Rodgers, and J. Davis. SCAPE: Shape comple-tion and animation of people. ACM Transactions onGraphics, 24:408–416, 2005.

    [2] P. J. Besl and N. D. McKay. A method for registrationof 3-d shapes. IEEE Transactions on Pattern Analysisand Machine Intelligence, 14(2):239–256, 1992.

    [3] C. M. Bishop. Neural Networks for Pattern Recogni-tion. Oxford University Press, Inc., New York, NY,USA, 1995.

    [4] H. Chui and A. Rangarajan. A feature registrationframework using mixture models. In Proceedings ofIEEE Workshop on Mathematical Methods in Biomed-ical Image Analysis, pages 190–197, 2000.

    [5] H. Chui and A. Rangarajan. A new point matchingalgorithm for non-rigid registration. Computer Visionand Image Understanding, 89(2-3):114–141, 2003.

  • [6] S. Ge, G. Fan, and M. Ding. Non-rigid point set reg-istration with global-local topology preservation. InProceedings of IEEE Conference on Computer Visionand Pattern Recognition Workshops, pages 245–251,2014.

    [7] R. Horaud, F. Forbes, M. Yguel, G. Dewaele, andJ. Zhang. Rigid and articulated point registration withexpectation conditional maximization. IEEE Trans-actions on Pattern Analysis and Machine Intelligence,33(3):587–602, 2011.

    [8] B. Jian and B. C. Vemuri. A robust algorithm for pointset registration using mixture of Gaussians. In Pro-ceedings of IEEE International Conference on Com-puter Vision, pages 1246–1251, 2005.

    [9] B. Jian and B. C. Vemuri. Robust point set registra-tion using Gaussian mixture models. IEEE Transac-tions on Pattern Analysis and Machine Intelligence,33(8):1633–45, 2011.

    [10] Y. Lipman, O. Sorkine, D. Cohen-Or, D. Levin,C. Rossi, and H.-P. Seidel. Differential coordinatesfor interactive mesh editing. In Proceedings of ShapeModeling Applications, pages 181–190, 2004.

    [11] J. Ma, J. Zhao, J. Tian, Z. Tu, and A. L. Yuille. Ro-bust esitimation of nonrigid transformation for pointset registration. In Proceedings of IEEE Conferenceon Computer Vision and Pattern Recognition, pages2147–2154, 2013.

    [12] J. Ma, J. Zhao, J. W. Tian, A. L. Yuille, and Z. W.Tu. Robust point matching via vector field consensus.IEEE Transactions on Image Processing, 23(4):1706–1721, 2014.

    [13] D. Mateus, R. Horaud, D. Knossow, F. Cuzzolin, andE. Boyer. Articulated shape matching using Laplacianeigenfunctions and unsupervised point registration. InProceedings of IEEE Conference on Computer Visionand Pattern Recognition, pages 1–8, 2008.

    [14] R. Merris. Laplacian matrices of graphs: a survey.Linear Algebra and its Applications, 197-198(0):143–176, 1994.

    [15] A. Myronenko and X. Song. Point set registration:Coherent point drift. IEEE Transactions on Pat-tern Analysis and Machine Intelligence, 32(12):2262–2275, 2010.

    [16] A. Myronenko, X. Song, and M. A. Carreira-Perpinan.Non-rigid point set registration: Coherent point drift(CPD). In Proceedings of Advances in Neural Infor-mation Processing Systems, pages 1009–1016, 2006.

    [17] V. Panaganti and R. Aravind. Robust nonrigid pointset registration using Graph-Laplacian regularization.In Proceedings of IEEE Winter Conference on Appli-cations of Computer Vision, pages 1137–1144, 2015.

    [18] S. Pellegrini, K. Schindler, and D. Nardi. A gener-alization of the ICP algorithm for articulated bodies.In Proceedings of British Machine Vision Conference,pages 87.1–87.10, 2008.

    [19] O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa,C. Rössl, and H.-P. Seidel. Laplacian surface edit-ing. In Proceedings of Eurographics/ACM SIG-GRAPH Symposium on Geometry Processing, pages175–184, 2004.

    [20] J. Tang, L. Shao, and X. Zhen. Robust point patternmatching based on spectral context. Pattern Recogni-tion, 47(3):1469–1484, 2014.

    [21] A. Weiss, D. Hirshberg, and M. J. Black. Home 3Dbody scans from noisy image and range data. In Pro-ceedings of IEEE International Conference on Com-puter Vision, pages 1951–1958, 2011.

    [22] J. Yang, J. P. Williams, Y. Sun, R. S. Blum, and C. Xu.A robust hybrid method for nonrigid image registra-tion. Pattern Recognition, 44(4):764–776, 2011.

    [23] M. Ye, X. Wang, R. Yang, R. Liu, and M. Pollefeys.Accurate 3D pose estimation from a single depth im-age. In Proceedings of IEEE International Conferenceon Computer Vision, pages 731–738, 2011.

    [24] M. Ye and R. Yang. Real-time simultaneous pose andshape estimation for articulated objects using a sin-gle depth camera. In Proceedings of IEEE Conferenceon Computer Vision and Pattern Recognition, pages2353–2360, 2014.

    [25] A. L. Yuille and N. M. Grzywacz. A mathematicalanalysis of the motion coherence theory. InternationalJournal of Computer Vision, 3(2):155–175, 1989.

    [26] Z. Zhang. Iterative point matching for registration offree-form curves and surfaces. International Journalof Computer Vision, 13(2):119–152, 1994.

    [27] Y. Zheng and D. Doermann. Robust point matchingfor nonrigid shapes by preserving local neighborhoodstructures. IEEE Transactions on Pattern Analysis andMachine Intelligence, 28(4):643–649, 2006.

    /ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 200 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 2.00333 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 400 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 600 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.00167 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (None) /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False

    /CreateJDFFile false /Description > /Namespace [ (Adobe) (Common) (1.0) ] /OtherNamespaces [ > /FormElements false /GenerateStructure false /IncludeBookmarks false /IncludeHyperlinks false /IncludeInteractive false /IncludeLayers false /IncludeProfiles true /MultimediaHandling /UseObjectSettings /Namespace [ (Adobe) (CreativeSuite) (2.0) ] /PDFXOutputIntentProfileSelector /NA /PreserveEditing false /UntaggedCMYKHandling /UseDocumentProfile /UntaggedRGBHandling /UseDocumentProfile /UseDocumentBleed false >> ]>> setdistillerparams> setpagedevice