Computational Framework for Performance Characterization of 3-D Reconstruction

Techniques from Sequence of Images

Ahmed Eid and Aly FaragComputer Vision and Image Processing (CVIP) Laboratory

University of Louisville, Louisville, KY 40292

URL http://www.cvip.uofl.edu

2004

2

www.cvip.uofl.edu

Problem Description

3-D ModelingTechniques

Testing Methodologies

Definition ofGround Truth

Quality Measures

Quantification of the performance of 3-D modeling techniques by designing and defining 4 dependent modules:

• Test setup• Ground truth• Testing methodology• Quality measures

Design of Test beds

3

www.cvip.uofl.edu

Notations

• Ground truth set G: a finite set of reference 3D, 2D, or 1D points describing a 3D object or its images

• Data set M: a finite set of points obtained with a given 3D reconstruction technique from the collected images

• Comparable set D: a set of data points transformed in order to be compared to the ground truth G using a relevant transformation C

• Generally, performance characterization may need an additional registration of the transformed set M and G: The registration function T is a mapping:

T : G G’ such that (M T(G) )2 min

4

www.cvip.uofl.edu

Motivations Objectives

• Many vision-based applications need accuracy validation.

• Lack of standard Evaluation methodologies and measures

• Difficulty of generating ground truth data

1- To introduce a model for performance evaluation systems

2- To build a database consisting of data acquired by our system

3- To provide a comparative study of different 3-D reconstruction techniques.

4- Validation, Evaluation and Design of vision-based systems.

5

www.cvip.uofl.edu

3-D reconstruction techniques

Stereo based approaches Volumetric approaches

Area based Feature based Shape from silhouettes

Voxel coloring Space carving

Generalized voxel coloring

Stereo Volumetric

3-D Reconstruction: An overview

6

www.cvip.uofl.edu

Test Setup

Rotating backgroundscanner head

CCD camera

scanner base

support

support

a- System components b- Top view

t

7

www.cvip.uofl.edu

Test Setup (cont.)

1- Sequence of images (discrete mode)

2- Panoramic images (continuous mode)

8

www.cvip.uofl.edu

Camera Calibration

• Compute the Projection Matrix P0 at the initial position using calibration pattern.

• Assumption: the camera rotates around Y-axis by an angle the projection matrix Pk at view k is:

where k=0, 2….., I1 (I is the number of acquired images)

1000

0kcos0ksin

0010

0ksin0kcos

P)(P 0k

9

www.cvip.uofl.edu

),(),( 1311 iiii ZXfpZXgpdx

dydx

pZXgpYpZXfp iiiii

Mcardi ,max

),(),(5.0 34333231

)(1min

),(),( 2321 iiii ZXfpZXgpdy

)sin()cos(),( ZXZXf

)cos()sin(),( ZXZXg

for 5.0,5.0 yx

Setup Accuracy

The error in the rotation angle is

Where,

10

www.cvip.uofl.edu

Background Subtraction• Acquire a sequence of background images

• Acquire a sequence of data images

• Subtraction model: normal p.d. of differences between color signals for the same background pixel

– zero mean and a diagonal covariance matrix to be estimated [Elgammal’ 00]

3

1j

xx25.0

jbd

2

j

jbjd

e2

1)xx(P

Use the color probability density estimate:

The pixel is considered a foreground one if P(xd ,xb) < T1 where T1 is a chosen threshold.

• Apply median filter to fill gaps

11

www.cvip.uofl.edu

Background Subtraction (cont.)

Example

T1 = 0.005

(a) input (b) background

(c) Background subtraction (d) After median

filtering

(e) Restore colors

12

www.cvip.uofl.edu

3-D Data RegistrationRegistration Through Silhouettes (RTS)

• Align ground truth G and the measured data M such that

i

ii GgTMm ))(,(dE 2

Where d denotes the distance

• Conventional 3-D registration techniques may fail in the evaluation problem because M may be corrupted set of data which or manual selection of matched points can be used.• The proposed RTS method doesn’t use the set M through the registration process however, it uses the silhouettes of the input images

13

www.cvip.uofl.edu

RTS Procedure

1- Generate Ns of input image silhouettes IM .

2- Generate Ns of ground truth silhouettes IG using the projection matrices at same views of input silhouettes and the registration parameters x, y, z, tx, ty, and tz.3- Compute the error criterion:

221

lsN

1l

hN

11k

wN

12k21

l

whs

))k,(kI)k,(k(INNN

1GM

4- Go to step 2, stop when is minimum.

Minimum error criterion indicates full alignment of input silhouettes and ground truth silhouettes. This results in registration of G and M.

14

www.cvip.uofl.edu

Two-step optimization

• We used the a genetic algorithm (GA) to minimize the error criterion. Since the GA maximizes an objective function, then we use

1F

• The used cross over rate is pc=0.95 and the mutation rate pm=0.01.• Since GA takes long time to converge to exact solution, we get only approximate by GA then use it as initial solution to a local search method.

Gene structure

To be maximized. We used the gene structure shown below.

15

www.cvip.uofl.edu

(a) input images at 0, 90o

(b) Corresponding silhouettes of (a)

(c) initial silhouettes of scanner (%10)

(d) After 100 iterations of GA (%10)

(e) After 150 iterations of simplex (%100)

3-D Registration (cont.)

16

www.cvip.uofl.edu

3-D Registration parameters

(a) X-translation (c) Z-translation(b) Y-translation

(d) X-rotation (e) Y-rotation (f) Z-rotation

17

www.cvip.uofl.edu

Registration Visual Results

(a) before (b) after

(c) standard

18

www.cvip.uofl.edu

• It assesses the performance in the 3-D domain• It decimates the registered 3-D data sets G’ and M into Nm batches.• It uses the local centroids and deviations from centroids of corresponding pair of patches to assess the quality.

The patch size is determined by:

Where Nm is a user defined parameter, 3m iN and i is a positive integer

)max( 'b M,Gu

)min( 'b M,Gl

and,

)(N

bbm

luΔX 32

1

Δz)Δy,x,(ΔΔX

,

Performance Evaluation Techniques

1- Local Quality Assessment (LQA) Test

19

www.cvip.uofl.edu

)C,d(CC j'GjM

jd

j'G

jMjD D

DR

Let C jd , is the centroid distance of patches Mj and G’j

and R jD , is the deviation ratio of patches Mj and G’j

Then similarity index Qj for the patches pair j is defined as:

max

jd

2jD

jDj

C

C1

R1

R2Q

222max ΔzΔyΔx2C Where,

Qj has range [0 1], with 1 represents the highest quality

20

www.cvip.uofl.edu

Example

Nm = 64

(a) bar graph of Q (b) histogram of Q in (a)

10 20 30 40 50 600

0.2

0.4

0.6

0.8

1

Data Subsets

Qua

lity

Inde

x (Q

)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

10

20

30

40

50

60

Quality Index (Q)

Per

cent

age

Cou

nt

21

www.cvip.uofl.edu

0 0.2 0.4 0.6 0.8 10

0.5

1

1.5

2

2.5

3

3.5

q

f Q(q)

=0.1,=0.1

=2,=2

=1,=3 =3,=1

1

11

)ab)(,(B

)qb()aq()q(f

dt)t(t)ˆ,ˆ(B

)qQ(Pq

ˆˆ

0

11 11

1

Probability Estimate of Quality

Let the pdf of beta distribution is defined as:

a=0, b=1 for q[0 1], and and are shaping parameters

Where ˆˆ and are the maximum likelihood estimators of beta distribution

The probability estimate P is defined as:

22

www.cvip.uofl.edu0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

q

Pq(Q

q

)

3618129

0

0.1

0.2

0.3

0.4

0.5

0.6

Nor

mal

ized

His

togr

am

0.05 0.15 0.25 0.35 0.45 0.55 0.65 0.75 0.85 0.95

Quality Index (Q)

Effect of No. of input images on Q

9 12 18 36

Results

Applied to Space Carving with varying number of input mages

(a) inputs=12

(c) Difference corresponds to (a)

(b) inputs=9

Fattening

23

www.cvip.uofl.edu

2- Image Re-projection (IR) Test

Performance Evaluation Techniques

• This test can be provided if the 3-D ground truth is not available• It is distinct from the [Szeliski’99] in the way it is not restricted to the stereo approaches and it uses calibrated data so there is no need for prediction procedures.• It uses quality measures that is used to assess the quality of real images.

1- Generate the image data set D of by projecting the data set M using projection matrices C at each view.

2- Compare D = C(M) to the original images G (as the ground truth) using a certain quality measure.

Test Procedure

24

www.cvip.uofl.edu

Quality Measures

1- Signal to Noise Ratio for N x M images:

j,i2

j,i2

10))j,i(d)j,i(g(

)j,i(glog10)dB(SNR

Here and below g(i,j) and d(i,j) are intensities at the pixel (i,j) of the reference and data images, respectively

Image Re-projection (IR) Test

25

www.cvip.uofl.edu

2- Quality Index Q [Z. Wang’02]

where g and d are the s.d. of reference and data images, respectively

g and d are the means of reference and data images, respectively

gd is the covariance between g and d

Q has dynamic range [-1 1] and can be modified to Qm as

2222

22

dg

dg

dg

gd

dgdg

Q

)(log)( QdBQm 210 10

Quality Measures (cont.)Image Re-projection (IR) Test

26

www.cvip.uofl.edu

3- Fuzzy Image Metric (FIM) [J. Li’02] based on differences

where Kcard(G)=card(D) and 0gi,di 1 after normalization

where V is the domain, ucard ({.})/K, u(V)1, and Nl(f)={x|f(x)>l }FIM has the dynamic range [0 1], and the related IFIM is

),........,( KK dgdgdgDG 2211

Quality Measures (cont.)

))((,/min(max

)()(

/ DGNui

duDGSDGFIM

ii

V

255

2550255

FIMdBIFIM

110 10log)(

Image Re-projection (IR) Test

27

www.cvip.uofl.edu

ResultsApplied to Space Carving at different resolutions =1.25, 1.67, 2.00 and 2.50 mm

No. of input images =36(a) original

(b) =1.25 (c) =1.67

(e) =2.5(d) =2

28

www.cvip.uofl.edu

Applied to Space Carving at different numbers of input images 36, 18, 12, and 9

Results

(a) inputs=12

(c) Difference corresponds to (a)

(d) Difference corresponds to (b)

(b) inputs=9

Fattening

29

www.cvip.uofl.edu

Parameter =1.25 mm =1.67 mm =2.00 mm =2.50 mm

initial No. of voxels 13997521 5929741 3442951 1771561

No. of Input Images =36

final No. of voxels 81535 48190 32361 20063

execution time (min.) 126.8 39.4 18.5 7.8

mean (dB) 9.190 7.632 6.497 4.727

No. of Input Images =18

final No. of voxels 83233 49629 33310 20459

execution time (min.) 55.7 21.1 21.1 3.9

mean (dB) 9.012 7.631 6.453 4.662

No. of Input Images =12

final No. of voxels 83076 49681 33305 20606

execution time (min.) 39.7 12.8 6.4 2.6

mean (dB) 9.000 7.402 6.445 4.652

No. of Input Images =9

final No. of voxels 87591 52906 35069 21519

execution time (min.) 26.8 9.1 4.3 1.8

mean (dB) 7.069 5.624 5.274 3.774

The effect of number of images on the quality of the output reconstructions and the executiontime at different resolutions.

30

www.cvip.uofl.edu

Comparative Study

It has two views:

1- Competitive view: Contrasts the differences between different approaches based on the proposed testing methodologies and measures. Different stereo techniques will be evaluated among themselves, then compared to the volumetric approaches

2- Cooperative view: Merge different techniques to achieve better quality. This integration depends on the outcome of the competition between algorithms. The LQA is suitable in this application

Post-evaluation

31

www.cvip.uofl.edu

An Example of stereo and space carving comparison

The IR test is applied to outputs of stereo and space carving on the same set of 12 input images.

Original Space Carving with true colors Stereo with true colors

Space Carving with consistent colors Stereo with consistent colors

32

www.cvip.uofl.edu

Results

SNR values at each view (12 images) for both space carvingand stereo: (a) resolution; (b) resolution + color consistency

(a) (b)

33

www.cvip.uofl.edu

Results

Fuzzy measure Quality index

34

www.cvip.uofl.edu

3-D Data Fusion

Given two reconstructions 1and 2 (have some dissimilarities) of an object, then we need to fuse them into a single reconstruction

1

2

1- The proposed method uses the contours generated from intensity images to guide the fusion process.

2- Corresponding contours are then generated from the reconstructions 1and 2

3- The evaluation methodology is applied.

4- The patches of low quality index are elected for the fusion process

The Fusion Procedure

35

www.cvip.uofl.edu

The Fusion Procedure (cont.)

1

W0

2 0

1

0

0

2

1

2

0

2

0

1

V0

3D reconstructions

projected contours

5- the corresponding contour segments of each low quality patch are determined

6- Two tests:(a) The closest point(b) The closest contour

are applied to the contour segments (from the given reconstructions) to determine which of them is closer to the reference contour segment

7- The surface patch that has closest contour segment to the reference one is included in the output reconstruction

36

www.cvip.uofl.edu

1

W0

2

0

1

0

0

2

)(,....,1 , jji cardip ii pcpc21

and

),(min),(

such that

1)

1(,...1

11

111

hi

jcardh

ri

ri

pcpcpcpcd

pcpc

For each point , the closest pointsare calculated as follows:

),(min),(

such that

2)

2(,...1

22

222

hi

jcardh

ri

ri

pcpcpcpcd

pcpc

and,

1- Closest Point Test

37

www.cvip.uofl.edu

2- Closest Contour Test

closer to is and 21

jjj To determine which of , the average distances

),( and ),(21

jjav

jjav dd are calculated as:

),(1

),(1

11 pcpd

)card(ξd

)jωcard(ξ

i

iiij

ω

jjav

),(1

),(1

22 pcpd

)card(ξd

)jωcard(ξ

i

iiij

ω

jjav

and

The fusion decision is taken based on which of ),( and ),(21

jjav

jjav dd

is minimum

38

www.cvip.uofl.edu

ResultsGiven reconstructions

(a) by 3-D laser scanner

(b) by Space Carving

39

www.cvip.uofl.edu

Results (cont.)

Silhouette from input imageSilhouette from Space Carving Silhouette from scanner

Contours from input image and space carving

Contours from input image and scanner

40

www.cvip.uofl.edu

Results (cont.)

Play video

Space Carving

Play video Laser scanner

Play videoFused reconstruction

41

www.cvip.uofl.edu

Validation of Reconstructions of the Human Jaw

Applications

• We plan use the evaluation framework to the validate the reconstructions of human jaw as a medical application.

• Two reconstruction approaches will be validated space carving and shape from shading with range data.

• Old method of validation used manual measurements from the real jaw then compared them to measurements on the reconstructed model

42

www.cvip.uofl.edu

The CardEye System at CVIP lab.

Evaluation of the CardEye Trinocular Active Vision System

• The system employs a sensor planning technique for multiple camera vision systems. This allows the automatic selection of camera parameters (vergence, baseline, zoom and focus) that satisfy different vision constraints: field of view, focus, disparity and overlap.

• The effect of the above parameters on the output reconstruction will be investigated.

• We will study the effect of adding the third camera on the robustness of the output data.

Applications

43

www.cvip.uofl.edu

Conclusions and Future Extensions• While 3-D reconstruction approaches are used in many applications there are no widely accepted methodologies of quantifying the performance of these approaches.• There is a real difficulty of having dense ground truth data.• Most of the evaluation work done so far is dedicated to stereo approaches.

In this study, we proposed the followings:1- A design of a test bed that is able to collect optical data and dense ground truth data. Related problems to the system design are investigated such as:

• Camera calibration. • System accuracy (we presented a closed form to system accuracy).• Data segmentation (we proposed a background subtraction technique).• Data registration (we proposed an efficient 3-D registration technique using silhouettes (RTS).

44

www.cvip.uofl.edu

Conclusions and Future Extensions2- Two testing methodologies:• A 3-D evaluation test, the (LQA) test, that can detect local errors in the 3-D model. Which makes it suitable for error analysis and fusion techniques.• A 2-D evaluation test, the (IR) test that uses input images as an implicit ground truth. The test is suitable for investigating the quality of the output for applications of virtual reality.3- We provided an example of evaluation the performance of stereo and space carving under the same framework.

In addition we propose the following extensions:1- Find a relation between the error distribution in 2-D and 3-D. The will validate the use of images as ground truth.2- Differentiate between the errors that caused by the 3-D technique itself or that is may be introduced by the testing methodology itself.3- Study object recognition techniques to find good description of an object. This will give good measured to be used in the LQA test.

45

www.cvip.uofl.edu

Conclusions and Future Extensions

4- Provide more comparisons between different reconstruction techniques.5- Enhance the quality of 3-D reconstructions using data integration techniques.6- Validate the results provided by human jaw modeling system7- Evaluate the performance of the Cardeye system.9- Design a 3-D passive scanner