Communication Systems Group Technische Universität Berlin S. Knorr A Geometric Segmentation...

Communication Systems Group

Technische Universität Berlin

S. Knorr

A Geometric Segmentation Approach for the 3D Reconstruction of Dynamic Scenes in 2D Video

Sequences

Technical University of BerlinCommunication Systems Group

Director: Prof. Thomas Sikora

Sebastian Knorr, Evren Imre*, A. Aydin Alatan*, and Thomas Sikora

* Middle East Technical UniversityEEE Department

Assoc. Prof. Aydin Alatan

EUSIPCO 2006 2Communication Systems GroupS. Knorr


Outline

• Motivation• Overview of proposed solution• Feature tracking and motion segmentation• Multiview 3D reconstruction• Simulation results • Future work



Motivation

S. Knorr, E. Imre, B. Özkalayci, A. A. Alatan, and T. Sikora, A Modular Scheme for 2D/3D Conversion of TV Broadcast, 3DPVT'06

E. Imre, S. Knorr, A. A. Alatan, and T. Sikora, Prioritized Sequential 3D Reconstruction in Video Sequences of Dynamic Scenes, ICIP'06



Motivation

This work:

• 3D reconstruction of dynamic scenes with independently moving objects (IMOs) using Structure from Motion (SfM) techniques

• Scenes are captured with a single camera

• Static background and IMOs are reconstructed independently

Main goal: 3D reconstruction of TV broadcast video



Examples



Overview of the Proposed Solution

Feature Detection & Tracking

Iterative F-Matrix Estimation &

Motion Segmentation

Initial Structure Computation

Prioritized Sequential Structure

Estimation



Outline




Feature tracking and motion segmentation

• Harris-Corner-Detector for feature selection

• pyramidal Lucas-Kanade tracker to track features along the whole sequence

• Geometric Robust Information Criterion (GRIC) for keyframe selection

kdneGRIC i 212

2D motion model, H (homography), for small baselines vs. 3D motion model, F (epipolar geometry), for large baselines

[P.H.S. Torr, A.W. Fitzgibbon and A. Zisserman (ICCV'98)]



Keyframe Selection

240-frame sequence (TUB-room) with 14 keyframes



Motion segmentation

• F-matrix estimation for consecutive keyframes (RANSAC and re-RANSAC) labeling of background and IMO trajectories

• Increasing the number of features on the IMO (guided-matching)

021 xFx iT

For each independent motion in the sequence, there exists a corresponding F-matrix, Fi, which

fulfills the epipolar constraint



Outline




Multiview 3D reconstruction

• Fast convergence to a reliable estimate:Since the quality of subsequent reconstructions depend on the current (intermediate) structure estimate, errors in the first few frame pairs may cause the entire estimation procedure to collapse.

• Fast recovery of the scene structure: The number of reconstructed 3-D points should be maximized, while processing a minimum number of frame pairs.

Main goals:



Prioritized Sequential Reconstruction

1. Compute the initial reconstruction and camera path (e.g. Pollefeys et al.).

2. Compute the priority metric and order the frame pairs.

Given the internal calibration parameters and the feature trajectories:

2-view rec.1:6

2-view rec.2:4

2-view rec.7:9

Add 5, 2-viewrec. 1:5


Merge sub-reconstruction

3:6


Merge sub-reconstruction

7:8

Two-view reconstruction

Sub-estimate fusion using 3D-2D correspondencesSub-reconstruction fusion using 3D-3D correspondences

Ordered pair list: 1:6, 2:4, 1:5, 3:4, 3:6, 7:9, 3:8, 7:8



Outline




Simulation Results (1)

frame 1

frame 170




frame 1

frame 200



Future Work

2D TV-Display

Autostereoscopic Display

Shutter-Glases

Anaglyph



Future Work

initial dense depth map refined depth map virtual right stereo view

original left stereo view



Future Work

original left stereo view virtual right stereo view



Thank you for your attention!

Questions?

?



Motion segmentation

• For each independent motion in the sequence, there exists a corresponding F-matrix, Fi, which fulfills the epipolar constraint

where x1 and x2 are corresponding points in two views.

• A RANSAC-based F-matrix estimation algorithm identifies the feature pairs belonging to the dominant motion and labels the rest as outliers.

• Some of the outliers should satisfy the epipolar constraint according to a new F-matrix, which corresponds to the motion of an independent moving object (IMO)

021 xFx iT



Segmentation Algorithm

1. Compute the F-matrix corresponding to the first and the second key-frame by using a RANSAC-based procedure and label the inliers as background trajectories.

2. Compute the F-matrix on the outliers of step 1 by using again RANSAC and label the inliers as IMO trajectories.

3. Compute the centroid of the inliers of step 2 and check their distances. If the distance is higher than a threshold, reject the feature.

4. Increase the number of features on the IMO in consecutive key-frames with guided-matching.

5. Repeat step 2 to 4 as long as the F-matrix estimation is still reliable and most of the remaining features are spatially close.

6. Proceed to the next key-frame. Estimate the F-matrix between the last and the current key-frame for each motion using the labeled trajectories and classify new trajectories using step 1 to 5.

7. Repeat step 6 for all key-frames.



Priority Metric

cnb

adp

exp1

The priority metric, p, utilized in the algorithm to evaluate the feasibility of a frame pair for reconstruction is defined as

where d is the baseline distance between the cameras, n the number of feature matches, a, b and c are the design parameters of the sigmoid function appearing in the second term. The non-linear (sigmoidal) weighting keeps the contribution of the second term within a bound, when there is a relatively small or large number of matching features.




# frame pairs

(used/total)

# 3D points

(recov./total)

reproj. error

Cliff 45 / 5778 5890 / 8212 0.95

Palace 25 / 19900 2771 / 3546 0.90

TUB-Room

17 / 28680 4716 / 6095 0.27



Segmentation Results

frame 83frame 1

frame 201frame 170

Communication Systems Group Technische Universität Berlin S. Knorr A Geometric Segmentation...

Documents

Transcript of Communication Systems Group Technische Universität Berlin S. Knorr A Geometric Segmentation...