Bryan Willimon, Steven Hickson, Ian Walker, and Stan Birchfield

Clemson UniversityIROS 2012 - Vilamoura, Portugal

An Energy Minimization Approach to 3D Non-Rigid Deformable Surface Estimation Using

RGBD Data

We propose an algorithm that uses energy minimization to estimate the current configuration of a highly non-rigid object.

Our approach relies on a 3D nonlinear energy minimization framework to solve for the configuration using a semi-implicit scheme adapted from Fua and colleagues (Pilet et al. 2005).

Overview

Previous Work on Pose Estimation for Robotics

• Elbrechter et al. (IROS 2011) use a soft-body-physics model with visual tracking to manipulate a piece of paper.

• Bersch et al. (IROS 2011) describe a method to bring a T-shirt into a desired configuration by alternately grasping the item with two hands, using a fold detection algorithm.

Both approaches require predefined fiducial markers.

The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms:

Smoothness term

Energy Minimization Approach

Correspondence term

Depth term

Boundary term

The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms:

Energy Minimization Approach

Mesh Initialization

Energy Minimization Approach

Energy Minimization Approach

Smoothness term

Correspondence term

Depth term

Boundary term

Smoothness term

Energy Minimization Approach

Correspondence term

Energy Minimization Approach

Depth term

Energy Minimization Approach

Front View Top View

Boundary term

Without BoundaryWith Boundary

Energy Minimization Approach

Energy Minimization Approach

Minimize energy equation

Experimental Results

• We captured RGBD video sequences of shirts and posters to test our proposed method’s ability to handle different non-rigid objects in a variety of scenarios.

Four experiments were conducted:

1) Illustrating the contribution of the depth term

2) Illustrating the contribution of the boundary term

3) Partial self-occlusion

4) Textureless shirt sequence

Experimental Results

• Illustrating the contribution of the depth term

Experimental Results

• Illustrating the contribution of the boundary term

Experimental Results

• Partial self-occlusion

Experimental Results

• Textureless shirt sequence

Experimental Results Video

ConclusionWe have presented an algorithm to estimate the 3D configuration of a highly non-rigid object through a video sequence using feature point correspondence, depth, and boundary information.

We plan to extend this research to handle a two-sided 3D triangular mesh that covers both the front and the back of the object.

Questions?