A Prototype System for 3D Dynamic Face Data Collection by Synchronized Cameras

Yuxiao Hu

Hao Tang

Problem Statement

Collect multi-view face video with expressions; Potential researches

Co-articulation of facial expression and lip movement: Non-frontal view audio/visual speech recognition-lip

reading

Relevant Works

Static 2D face databases: FERET, CMU PIE, ORL, Yale Database, UMIST, etc

Static 3D face databases: 3D-RMA, GavabDB, YorkDB, XM2VTS database, FRGC database,etc

3D Dynamic face databases: CMU FIA, no markers, no audio Intel Research China Database, not synchronized

Highlights

Total Solution: Both hardware and software MultiView+Synchronization+RealTime Flexibility: Flexibly extended from 2 cameras to 5 ca

meras; Supplementary Tools:

camera calibration, color space conversion, 2D facial feature tracking 3D face shape recovery

Physical Setup

FoamHead

System Diagram

Camera Calibration

Video Data Capture

Color De-mosaicing

Facial Feature Tracking

3D Shape Reconstruction

Synchronization-Hardware Configuration

DragonFly Camera

Y

Synchronization-Software Implementation…

Buffers

BufferOverrun?

BufferOverrun?

BufferOverrun?

BufferOverrun?

Time StampMatched?Time StampMatched?

…Y Y

Y

N

NRe-syncRe-sync

CompressionCompression AVIAVI

Offline Color De-mosaic

Raw Data: Color represented in Sparse (Stippled) Pattern

Reconstructed RGBcolor image

Raw Data Reconstructed RGBcolor video

Camera Calibration

Find the intrinsic and extrinsic parameters Use Camera Calibration Toolbox for Matlab Two-step procedure

Find projection matrix using Direct Linear Transformation

Use as initialization for nonlinear minimization of mean squared re-projection error

Camera Calibration (cont’ed)

Camera 1 Camera 2

Camera 1

Camera 2

Camera Calibration (cont’ed)

Camera Calibration (cont’ed)

Camera 1

Camera Calibration (cont’ed)

-0.6 -0.4 -0.2 0 0.2 0.4

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

Reprojection error (in pixel)

x

y

-0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

Reprojection error (in pixel)

x

y

Camera 1 Camera 2

Average re-projection error < 0.2 pixels(0.16279,0.13482) and (0.16439,0.12685)

Facial Marker Tracking

Simple but effective tracking algorithm

Facial Marker Tracking (cont’ed) Statistical marker collocation model

Facial Marker Tracking (cont’ed)

Facial Marker Tracking (cont’ed)

3D Reconstruction: Stereo Triangulation A bit of theory

3D Reconstruction: Stereo Triangulation (cont’ed)

Deliveries

The data acquisition system of camera array Tools for Color De-mosaicing The calibration data and tools Some sample data result 3D ground truth data and labeling tool Technical Report

Outline (4Ws+2Hs)

Why (do we do this?) Who (has done the related work?) What (we proposed to do?) How (did we achieve our goal?) Why (we need to do so?) How (we evaluate our work?)