MMV Research Laboratory :A Retrospective Around Multimedia and Computer Vision Projects

MMV Research Laboratory: A

Retrospective Around Multimedia and

Computer Vision Projects Ivan Cabezas

[email protected]

July 18th 2012

Universidad Señor de Sipán – Chiclayo, Peru

Content

Universidad del Valle A Brief in Figures

Multimedia and Vision Laboratory National Cooperation

Industrial Collaboration

International Cooperation

Research Interests

A Camera Model

Some Research Projects

MPEG7 - SOS

Char Morphology

An Evaluation Methodology for Stereo Correspondence Algorithms

Final Remarks

MMV Research Laboratory: A Retrospective Around Multimedia and Computer Vision Projects

The Universidad del Valle

The Universidad del Valle is the largest university in the south west of

Colombia


http://www.univalle.edu.co

Universidad del Valle: A Brief in Figures

Its main campus, Meléndez, has an extension of a million of square meters

There are two campus in Cali, and nine regionals in Valle and Cauca

There are 187 study programs offered in Cali, most of them for graduate

There are six faculties and two institutes

At February of 2012, it had a population of 27094 students

(88.7% undergraduate)

At December of 2011, it had 889 full time professors

(92% graduate, 30% PhD)



Multimedia and Vision Laboratory

MMV is a multidisciplinary research group of the EISC

Ivan at WAC

2012

LACNEM, 2009


Meetings, 2007 & 2011

Maria at UNAL

2011

INTERACTIVIA, 2009

http://www.lacnem.org

National Cooperation

John W. Branch, UNAL- Medellín

Cesar Collazos, UniCauca - Popayán

Fabio González, UNAL - Bogotá

Liliana Salazar

Escuela de Ciencias Básicas

Doris Hinestroza

Departamento de Matemáticas

Juan Barraza

Escuela de Ingeniería Química

Janet Sanabria, Escuela de Recursos Naturales y

del Ambiente


Industrial Collaborations


International Cooperation

Ebroul Izquierdo Head of the Multimedia

and Vision Research Group, School of

Electronic Engineering and Computer

Science, Queen Mary University of London

Aggelos Katsaggelos, Director Motorola

Center for Seamless Communications,

Northwestern University, USA

Panos Liatsis, Head of the Information

Engineering and Medical Imaging Group,

School of Engineering and Mathematical

Sciences, City University London

Sergio Velastin, Director Digital Imaging

Research Centre, Kingston University, UK

Valia Guerra, Instituto de Cibernética,

Matemática y Física (ICIMAF), Cuba


Research Interests

Multimedia and Computer Vision


Computing System

Images

Information

Computer Vision http://www.slideshare.net/mmv-lab-univalle http://vision.mas.ecp.fr/Personnel/teboul/index.php/

A Camera Model

A Camera is a sensor following a model


http://quarknet.fnal.gov/fnal-uc/quarknet-summer-research/QNET2010/Astronomy/ http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/FUSIELLO4/tutorial.html

Camera System

3D World

2D Images


Some Research Projects

MPEG-7 UV

MPEG-7 SOS

Prokaryota

Vitisoft

Clusters: Espacial +

K - Means


MPEG-7 SOS: Motivation

M. Florian and M. Trujillo, Relational Database Schema for MPEG-7 Visual Descriptors, IEEE CBIR, 2008

M. Florian and M. Trujillo, Resource Oriented Architecture for Managing Multimedia Content, LACNEM, 2009

M. Florian MPEG-7 Service Oriented System, Master Research Project, Universidad del Valle, 2008


How to retrieve images stored over large (and distributed) repositories?

MPEG-7 SOS: Problem Statement


http://cs.usu.edu/htm/REU-Current-Projects


CBIR systems have some weaknesses

Annotations: wild life, horses,

chevaux, potros …

MPEG-7 SOS: MPEG-7 Standard



MPEG-7 SOS: The Proposal



Char Morphology

Char

Resin

Camera Microscopy

Particle Classification

1 Crassisphere

2 Inertoid

…

9 Mineroid


D. Chaves and M. Trujillo Impacto del Muestreo en la Clasificación de Carbonizados de Carbón, 5 CCC, 2010

Char Morphology: Motivation


Energy generation based on coal

http://www.iea.org/textbase/nppdf/free/2010/key_stats_2010.pdf http://www.worldcoal.org/coal/where-is-coal-found/

Char Morphology: Inherent Problems &

Proposed Approach


D. Chaves and M. Trujillo Impacto del Muestreo en la Clasificación de Carbonizados de Carbón, 5 CCC, 2010

Manual coal classification is a subjective and resources consuming process

Automatic Classification

Char Morphology: Inherent Problems &

Proposed Approach (ii)


D. Chaves and M. Trujillo Identificación Automática de Imágenes de Carbonizado Borrosas y con poco contenido, CONICA, 2012

Sampling and blurred or images with no content has to be considered

An Evaluation Methodology for

Stereo Correspondence Algorithms

Ivan Cabezas, Maria Trujillo and Margaret Florian [email protected]

February 25th 2012

International Conference on Computer Vision Theory and Applications, VISAPP 2012, Rome - Italy

Stereo Vision

The stereo vision problem is to recover the 3D structure of the scene using

two or more images

3D Model Stereo Images

Disparity Map

Left Right

Correspondence Algorithm

Reconstruction Algorithm

Camera System

3D World

2D Images

Inverse Problem

Optics Problem

Yang Q. et al., Stereo Matching with Colour-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling, IEEE PAMI 2009

An Evaluation Methodology for Stereo Correspondence Algorithms, VISAPP 2012, Rome - Italy

Canonical Stereo Geometry and Disparity

Disparity is the distance between corresponding points

Trucco, E. and Verri A., Introductory Techniques for 3D Computer Vision, Prentice Hall 1998


Accurate Estimation Inaccurate Estimation

P

Cr Cl

πl πr

pl pr

B

f

Z’

pr’

P ’

P

Cr Cl

πl πr

pl pr

B

f

Z

Ground-truth Based Evaluation

Ground-truth based evaluation is based on the comparison using disparity

ground-truth data

Scharstein, D. and Szeliski, R., High-accuracy Stereo Depth Maps using Structured Light, CVPR 2003

Tola, E., Lepetit, V. and Fua, P., A Fast Local Descriptor for Dense Matching, CVPR 2008

Strecha, C., et al. On Benchmarking Camera Calibration and Multi-View Stereo for High Resolution Imagery, CVPR 2008

http://www.zf-usa.com/products/3d-laser-scanners/


Quantitative Evaluation Methodologies

Szeliski, R., Prediction Error as a Quality Metric for Motion and Stereo, ICCV 2000

Kostliva, J., Cech, J., and Sara, R., Feasibility Boundary in Dense and Semi-Dense Stereo Matching, CVPR 2007

Tomabari, F., Mattoccia, S., and Di Stefano, L., Stereo for robots: Quantitative Evaluation of Efficient and Low-memory Dense Stereo Algorithms, ICCARV 2010

Cabezas, I. and Trujillo M., A Non-Linear Quantitative Evaluation Approach for Disparity Estimation, VISAPP 2011


The use of a methodology allows to:

Assert specific components and procedures

Tune algorithm's parameters

Support decision for researchers and

practitioners

Measure the progress on the field

Middlebury’s Methodology


Select Test Bed Images Select Error Criteria

Select Error Measures

nonocc all disc

Select and Apply Stereo Algorithms

Compute Error Measures

ObjectStereo GC+SegmBorder PUTv3

PatchMatch ImproveSubPix OverSegmBP


Scharstein, D. and Szeliski, R., http://vision.middlebury.edu/stereo/eval/, 2012

Middlebury’s Methodology (ii)




Algorithm nonocc all disc

ObjectStereo 2.20 1 6.99 2 6.36 1

GC+SegmBorder 4.99 6 5.78 1 8.66 5

PUTv3 2.40 2 9.11 6 6.56 2

PatchMatch 2.47 3 7.80 3 7.11 3

ImproveSubPix 2.96 4 8.22 4 8.55 4

OverSegmBP 3.19 5 8.81 5 8.89 6

Algorithm Average

Rank

Final

Ranking

ObjectStereo 1.33 1

PatchMatch 3.00 2

PUTv3 3.33 3

GC+SegmBorder 4.00 4

ImproveSubPix 4.00 5

OverSegmBP 5.33 6

Apply Evaluation Model


ObjectStereo 2.20 6.99 6.36

GC+SegmBorder 4.99 5.78 8.66

PUTv3 2.40 9.11 6.56

PatchMatch 2.47 7.80 7.11

ImproveSubPix 2.96 8.22 8.55

OverSegmBP 3.19 8.81 8.89


Middlebury’s Methodology (iii)


Scharstein, D. and Szeliski, R., A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms, IJCV 2002


Apply Evaluation Model Interpret Results

The ObjectStereo algorithm produces accurate results

Middlebury’s Evaluation Model

Algorithm Average

Rank

Final

Ranking

ObjectStereo 1.33 1

PatchMatch 3.00 2

PUTv3 3.33 3

GC+SegmBorder 4.00 4

ImproveSubPix 4.00 5

OverSegmBP 5.33 6

Middlebury’s Methodology (iv): Weaknesses


The Middlebury’s evaluation model have some shortcomings

In some cases, the ranks are assigned arbitrarily

The same average ranking does not imply the same performance (and

vice versa)

The cardinality of the set of top-performer algorithms is a free parameter

It operates values related to incommensurable measures

Middlebury’s Methodology (v): Weaknesses

The BMP percentage measures the quantity of disparity estimation errors

exceeding a threshold


The BMP measure have some shortcomings:

It is sensitive to the threshold selection

It ignores the error magnitude

It ignores the inverse relation between depth and disparity

It may conceal estimation errors of a large magnitude, and, also it may

penalise errors of small impact in the final 3D reconstruction

Cabezas, I., Padilla, V., and Trujillo M., A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011

Gallup, D., et al. Variable Baseline/Resolution Stereo, CVPR, 2008

The A* evaluation methodology brings a theoretical background for the

comparison of stereo correspondence algorithms

The set of algorithms under evaluation

The set of estimated maps to be compared

The function that produces a vector of error measures

The set of vectors of error measures

A* Methodology



A* Methodology (ii)



The evaluation model of the A* methodology addresses the comparison of

stereo correspondence algorithms as a multi-objective optimisation problem

It defines a partition over the set A (the decision space)

Subject to:

where ≺ denotes the Pareto Dominance relation:

Let p and q be two algorithms

Let Vp and Vq be a pair of vectors belonging to the objective space

Thus, three possible relations are considered

A* Methodology (iii): Pareto Dominance


Van Veldhuizen, D., et al., Considerations in Engineering Parallel Multi-objective Evolutionary Algorithms, Trans in Evolutionary Computing 2003

The Pareto Dominance defines a partial order relation

VGC+SegmBorder = < 50.48, 64.90, 24.33>

VPatchMatch = < 49.95, 261.84, 32.85>

VImproveSubPix = < 50.66, 97.94, 32.01>

VGC+SegmBorder VPatchMatch

< 50.48, 64.90, 24.33> < 49.95, 261.84, 32.85>

GC+SegmBorder ~ PatchMatch

VGC+SegmBorder VImproveSubPix

< 50.48, 64.90, 24.33> < 50.66, 97.94, 32.01>

GC+SegmBorder ≺ ImproveSubPix

A* Methodology (iv): Illustration




nonocc all disc







A* Methodology (v): Illustration

The evaluation model performs the partitioning and the grouping of stereo

algorithms under evaluation, based on the Pareto Dominance relation




ObjectStereo 2.20 6.99 6.36

GC+SegmBorder 4.99 5.78 8.66

PUTv3 2.40 9.11 6.56

PatchMatch 2.47 7.80 7.11


OverSegmBP 3.19 8.81 8.89

Algorithm nonocc all disc Set

GC+SegmBorder 50.48 64.90 24.33 A*

PatchMatch 49.95 261.84 32.85 A*

PUTv3 99.67 333.37 53.79 A’

ImproveSubPix 50.66 97.94 32.01 A’

OverSegmBP 58.65 108.60 34.58 A’

ObjectStereo 73.88 117.90 36.25 A’


, GC+SegmBorder PatchMatch

ObjectStereo PUTv3 ImproveSubPix OverSegmBP , , ,

A* Methodology (vi): Illustration

Interpretation of results is based on the cardinality of the set A*



The GC+SegmBorder and the PatchMatch algorithms are, comparable among them,

and have a superior performance to the rest of algorithms

A* Evaluation Model

Algorithm nonocc all disc Set

GC+SegmBorder 50.48 64.90 24.33 A*

PatchMatch 49.95 261.84 32.85 A*

ImproveSubPix 50.66 97.94 32.01 A’

OverSegmBP 58.65 108.60 34.58 A’

ObjectStereo 73.88 117.90 36.25 A’

PUTv3 99.67 333.37 53.79 A’

A* Methodology (vii): Strength and Weakness



Strength: It allows a formal interpretation of results, based on the cardinality

of the set A*, and in regard to considered imagery test-bed

Weakness: It does not allow an exhaustive evaluation of the entire set of

algorithms under evaluation

It computes the set A* just once, and does not bring information about A’

A* Groups Methodology


It extends the evaluation model of the A* methodology, incorporating the

capability of performing an exhaustive evaluation

It introduces the partitioningAndGrouping algorithm

A = Set ( { } );

A.load( “Algorithms.dat” );

A* = Set ( { } );

A’ = Set ( { } );

group = 1;

do {

computePartition( A, A*, A’, g, ≺ );

A*.save ( “A*_group_”+group );

group++;

A.update ( A’ ); // A = A / A*

A*.removeAll ( ); // A* = { }

A’.removeAll ( ); // A’ = { }

}while ( ! A.isEmpty ( ) );

subject to:

The A* Groups methodology uses the Sigma-Z-Error

(SZE) measure

The SZE measure has the following properties:

It is inherently related to depth reconstruction in a stereo system

It is based on the inverse relation between depth and disparity

It considers the magnitude of the estimation error

It is threshold free

A* Groups Methodology (ii): Sigma-Z-Error


Cabezas, I., Padilla, V., and Trujillo M., A Measure for Accuracy Disparity Maps Evaluation, CIARP 2011

A* Groups Methodology (iii): Illustration

The evaluation process of selected algorithms by using the proposal




nonocc all disc





A* Groups Methodology (iv): Illustration

The evaluation model performs the partitioning and the grouping of stereo

algorithms under evaluation, based on the Pareto Dominance relation




ObjectStereo 73.88 117.90 36.25

GC+SegmBorder 50.48 64.90 24.33

PUTv3 2.40 9.11 6.56

PatchMatch 49.95 261.84 32.85


OverSegmBP 58.65 108.60 34.58

, GC+SegmBorder PatchMatch


Algorithm nonocc all disc Group

GC+SegmBorder 50.48 64.90 24.33 1

PatchMatch 49.95 261.84 32.85 1

PUTv3 99.67 333.37 53.79


OverSegmBP 58.65 108.60 34.58

ObjectStereo 73.88 117.90 36.25


,

A* Groups Methodology (v): Illustration




PUTv3 99.67 333.37 53.79


OverSegmBP 58.65 108.60 34.58

ObjectStereo 73.88 117.90 36.25



ImproveSubPix 50.66 97.94 32.01 2

PUTv3 99.67 333.37 53.79

ObjectStereo 73.88 117.90 36.25

OverSegmBP 58.65 108.60 34.58

ImproveSubPix

ObjectStereo PUTv3 OverSegmBP , ,

ObjectStereo PUTv3 OverSegmBP , ,


PUTv3 99.67 333.37 53.79

OverSegmBP 58.65 108.60 34.58

ObjectStereo 73.88 117.90 36.25

ObjectStereo

OverSegmBP

, PUTv3


OverSegmBP 58.65 108.60 34.58 3

PUTv3 99.67 333.37 53.79

ObjectStereo 73.88 117.90 36.25

And so on …

A* Groups Methodology (vi): Illustration

Interpretation of results is based on the cardinality of each group



There are 5 groups of different performance

The GC+SegmBorder and the PatchMatch algorithms are, comparable among them,

and have a superior performance to the rest of algorithms

The ImproveSubPix algorithm is superior to

the OverSegmBP, the ObjectStereo, and the PUTv3 algorithms

…

The PUTv3 algorithm has the lowest performance

A* Groups Evaluation Model


GC+SegmBorder 50.48 64.90 24.33 1

PatchMatch 49.95 261.84 32.85 1

ImproveSubPix 50.66 97.94 32.01 2

OverSegmBP 58.65 108.60 34.58 3

ObjectStereo 73.88 117.90 36.25 4

PUTv3 99.67 333.37 53.79 5

Experimental Results

The conducted evaluation involves the following elements:


Test Bed Images

Error Criteria

Evaluation Models

Error Measures

A* Groups Middlebury

SZE , BMP

nonocc , all , disc


Stereo Algorithms 112 algorithms from the Middlebury’s repository

Experimental Results (ii)


Algorithm Strategy Group Middlebury’s

Ranking

DoubleBP Global 1 4

PatchMatch Local 1 11

GC+SegmBorder Global 1 13

FeatureGC Global 1 18

Segm+Visib Global 1 29

MultiresGC Global 1 30

DistinctSM Local 1 34

GC+occ Global 1 67

MultiCamGC Global 1 68

Algorithm Group Middlebury’s

Ranking

ADCensus 2 1

AdaptingBP 2 2

CoopRegion 2 3

DoubleBP 1 4

RDP 2 5

OutlierConf 2 6

SubPixDoubleBP 2 7

SurfaceStereo 2 8

WarpMat 2 9

ObjectStereo 2 10

PatchMatch 1 11

Undr+OverSeg 2 12

GC+SegmBorder 1 13

InfoPermeable 2 14

CostFilter 2 15

Conclusions

The use of the A* Groups methodology allows to perform an exhaustive

evaluation, as well as an objective interpretation of results

Innovative results in regard to the comparison of stereo correspondence

algorithms were obtained using proposed methodology and the SZE error

measure

The introduced methodology offers advantages over the conventional

approaches to compare stereo correspondence algorithms

Authors are already working in order to provide to the research community an

accessible way to use the introduced methodology


An Evaluation Methodology for

Stereo Correspondence Algorithms

Ivan Cabezas, Maria Trujillo and Margaret Florian [email protected]

February 25th 2012

International Conference on Computer Vision Theory and Applications, VISAPP 2012, Rome - Italy

Final Remarks

More information about the MMV-Lab can be found at

http://www.slideshare.net/mmv-lab-univalle

We are looking forward to create bounds with international collaborators

We invite you to participate at the 4th Latin American Conference on

Networked and Electronic Media, LACNEM, in Chile next October

If you have any question or concern please do not hesitate to contact me

[email protected] / www.ivancabezas.com


MMV Research Laboratory: A

Retrospective Around Multimedia and

Computer Vision Projects Ivan Cabezas

[email protected]

July 18th 2012

Universidad Señor de Sipán – Chiclayo, Peru

MMV Research Laboratory :A Retrospective Around Multimedia and Computer Vision Projects

Education

Transcript of MMV Research Laboratory :A Retrospective Around Multimedia and Computer Vision Projects