Image Processing and Analysis: Applications and …tavares/downloads/publications/...Image...

Image Processing and Analysis: Applications and Trends

João Manuel R. S. Tavares

[email protected] www.fe.up.pt/~tavares

AES ATEMA’ 2010 (Le Quebec, CANADA) Int. ConferenceMontreal & Quebec City, CANADA

June 27 to July 03 , 2010on

Advances and Trends in Engineering Materialsand their Applications

Outline

1. Introduction

2. Segmentation

3. Motion Tracking

4. Analysis of Objects: Matching, Registration and Morphing

5. 3D Reconstruction

6. Conclusions

7. Research Team

2Image Processing and Analysis: Applications and TrendsJoão Manuel R. S. Tavares

Introduction

• The researchers of Computational Vision aim the development of algorithms to perform in fully or semi-automatically manner operations and tasks carry out by the (quite complex) human’s vision system

4Image Processing and Analysis: Applications and Trends

Original images

Azevedo et al. (2010), Three-dimensional reconstruction and characterization of human external shapes from two-dimensional images using volumetric methods, Computer Methods in Biomechanics and Biomedical Engineering 13(3): 359-369

Computational 3D model built voxelized and poligonized

Introduction


• Image processing and analysis are topics of the most importance for our Society

• Algorithms of image processing and analysis are frequently used, for example, in:

– Medicine– Biology– Industry– Natural Sciences– Engineering

• Examples of common tasks involving algorithms of image processing and analysis are:

– noise removal– geometric correction– segmentation, recognition (2D-4D)– motion tracking and analysis, including matching, registration and morphing (2D-4D)– 3D reconstruction


Introduction


Introduction: Usual Computational Pipeline for Image Processing and Analysis

Image Processing and Analysis: Applications and Trends 6

Image(s) enhancement

Image(s) segmentation / features extraction

tracking

matching

morphing

Image(s)

motionanalysis registration

image processing

image analysis /computational

visionJoão Manuel R. S. Tavares

3D vision

computer vision

Segmentation

Segmentation• It is intended to identify in a full or semi- automatic manner

objects (2D/3D) presented in static images or in image sequences

• The most usual methodologies are based on template matching, statistical, geometric or physical modeling, or neuronal networks

• It is one of the most usual operations involved in the computational analysis of objects from images, and very often it is the first “important” step of image processing and analysis

• Frequent problems: noise, low resolution, reduce contrast, shapes not previously known, occlusion, multiple objects, etc.

João Manuel R. S. Tavares Image Processing and Analysis: Applications and Trends 8

Segmentation• Example: segmentation of contours in dynamic

pedobarography (Otsu method, morphologic operators)


Original images After segmentation

Bastos & Tavares (2004), Improvement of Modal Matching Image Objects in Dynamic Pedobarography using Optimization Techniques, Lecture Notes in Computer Science 3179:39-50

camera mirror

contact layer + glass

reflected light glass

pressure opaque layer

lamp

lamp transparent layer

Segmentation• Example: analysis of damage in composite materials

(binarization and region analysis)


Original image After segmentation

Damage areaMeasures obtained

Marques et al. (2009), Delamination Analysis of Carbon Fibre Reinforced Laminates: Evaluation of a Special Step Drill , Composites Science and Technology, 69(14): 2376-2382

Segmentation• Example: hardness evaluation from indentation images

(Johannsen and Bille threshold, region growing)


Vickers hardness Brinell hadnessFilho et al. (2010), Brinell and Vickers Hardness Measurement using Image Processing and Analysis Techniques, Journal of Testing and Evaluation 38(1):88-94

Segmentation• Example: analysis of material microstructures (neuronal

network)


Original images After segmentationAlbuquerque et al. (2008), A New Solution for Automatic Microstructures Analysis from Images Based on a Backpropagation Artificial Neural Network, Nondestructive Testing and Evaluation 23(4):273-283

Segmentation• Example: evaluation of nickel alloy secondary phases from

Scanning Electron Microscopic images (neuronal network)


Original image Image segmented

Albuquerque et al. (2010), Automatic Evaluation of Nickel Alloy Secondary Phases from SEM Images, Microscopy Research and Technique, doi: 10.1002/jemt.20870 (in press)

Segmentation• Example: evaluation on synthetic material porosity from

optical microscopic images (neuronal network)


Original images withtraining pixels

Image segmented

Albuquerque et al. (2010), New computational solution to quantify synthetic material porosity from optical microscopic images, Journal of Microscopy, doi: 10.1111/j.1365-2818.2010.03384.x (in press)

Segmentation• Example: control of a servomechanism by

gesture recognition (orientation histograms)


Tavares et al. (2005), Control a 2-Axis Servomechanism by Gesture Recognition using a GenericWebCam, International Journal of Advanced Robotic Systems 2(1):39-44

Running mode

Conversion for 256 gray levels

Orders orientation histogram vectors

initialization (stored in memory)

1 2 3 4

Orders images acquisition (one by one)

Command image

acquisition Conversion for 256

gray levels

Comparison of the images orientation histogram vector

with the stored vectors of the preset orders images

Associated command

order

1 3 4

1 2 4

1 2 3

2 3 4

4

2

1

Learning mode

Segmentation• Example: detection of breast tumours from mammography

images (Hough transform)


Chagas et al. (2007), An Application of Hough Transform to Identify Breast Cancer in Images, VIPimage 2007, 363-368

Original image After segmentation

Segmentation• Example: object recognition (image template matching)


Carvalho & Tavares (2005), Metodologias para identificação de faces em imagens: Introdução e exemplos de resultados, CMNI 2005

×fft fft

ift

( )3ift D CC( )2ift D CC

max CC

Original imageTemplate image

Segmentation• Example: segmentation of facial features

(deformable geometric template)


Carvalho & Tavares (2006), Two Methodologies for Iris Detection and Location in Face Images, CompIMAGE 2006, 129-134Carvalho & Tavares (2007), Eye detection using a deformable template in static images, VipIMAGE 2007, 209-215

Original image and associated energy fields

Segmentation of the iris using adeformable template (a circle)

Segmentation of an eye using an

deformable template

Segmentation• Example: segmentation of skin regions in images (statistical

model)


Skin samples used to build the statistical model Original image and

segmentation obtained

Carvalho & Tavares (2005), Metodologias para identificação de faces em imagens: Introdução e exemplos de resultados, CMNI 2005

Probably functionused


Segmentation• Example: segmentation of scene background/foreground in

image sequences (statistical models)

Backgroundsubtraction

Foreground objectdetection

Vasconcelos & Tavares (2008), Image Segmentation for Human Motion Analysis: Methods and Applications, WCCM8 / ECCOMAS 2008

Original images

Segmentation• Example: segmentation of faces and hands in images

(Active Shape Model)


Segmentations achieved (initial, intermediate and final steps)Vasconcelos & Tavares (2008), Methods to Automatically Built Point Distribution Models for Objects like Hand Palms and Faces Represented in Images, Computer Modeling in Engineering & Sciences 36(3):213-241

Segmentation• Example: segmentation of faces in images (Active Appearance

Model)


Vasconcelos & Tavares (2008), Methods to Automatically Built Point Distribution Models for Objects like Hand Palms and Faces Represented in Images, Computer Modeling in Engineering & Sciences 36(3):213-241

Segmentations achieved (initial, intermediate and final steps)

Segmentation• Example: analysis of the vocal tract shape during speech

production from MRI (Active Shape Models)


Vasconcelos et al. (2010), Towards the Automatic Study of the Vocal Tract from Magnetic Resonance Images, Journal of Voice, doi:10.1016/j.jvoice.2010.05.002 (in press)

Intermediate segmentation II

Originalimage

Finalsegmentation

Intermediate segmentation I

Segmentation• Example: analysis of the vocal tract shape during speech

production from MRI (Active Appearance Models)


Vasconcelos et al. (2010), Towards the Automatic Study of the Vocal Tract from Magnetic Resonance Images, Journal of Voice, doi:10.1016/j.jvoice.2010.05.002 (in press)

Intermediate segmentations

Initialsegmentation

Finalsegmentation

Intermediate segmentations

Segmentation• Example: segmentation of medical images (active contours -

snakes)


Original image andinitial contour

Final contour

Tavares et al. (2009), Computer Analysis of Objects’ Movement in Image Sequences: Methods and Applications, International Journal for Computational Vision and Biomechanics 2(2): 209-220

Segmentation• Example: segmentation of objects in images (deformable

contour, FEM, Lagrange equation)


Original images and initial contours Final contoursGonçalves et al. (2008), Segmentation and Simulation of Objects Represented in Images using Physical Principles, Computer Modeling in Engineering & Sciences 32(1):45-55

rubberk = 200N/m14s

Segmentation• Example: segmentation of medical images (level-set method)


Original image Initial segmentation Final segmentation

Perdigão et al. (2005), Geração de modelos de malhas de elementos finitos a partir de imagens médicas 2D, Encontro_1_Biomecânica, 81-85

Segmentation• Example: segmentation of pelvic floor from MRI (level-set

method, prior knowledge, shape influence field)


Ma et al. (2010), A shape guided C-V model to segment the levator ani muscle in axial magnetic resonance images, Medical Engineering & Physics, doi:10.1016/j.medengphy.2010.05.002 (in press)

Pelvic floor segmented

Segmentation• Example: new computational framework for medical image

segmentation (VC++, OpenCV, ITK)


Interface of our platform

Ma et al. (2008), Segmentation of Structures in Medical Images: Review and a New Computational Framework, CMBBE2008

Segmentation• Example: segmentation of organs from female pelvic cavity

MRI (using our platform)


Ma et al. (2010), A Review of Algorithms for Medical Image Segmentation and their Applications to the Female Pelvic Cavity, Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246

Region Growing Watershed

Malladi’s methodGeodesic Active Contour

Motion Tracking

Motion Tracking• It is intended to track the motion (and/or the deformation) of

objects along image sequences• In this area, the methodologies based on optical flow, block

matching and stochastic methods are widespread• Usually, it concerns the estimation of the motion involved,

the management of the features being tracked, and the analysis of the motion tracked as well as its quantification

• Usual problems: non-rigid motions, geometric distortions, non-constant illumination, occlusions, noise, multiple objects, etc.


Motion Tracking• Computational framework to

track features in image sequences(Kalman Filter or Unscented KalmanFilter, optimization, Mahalanobisdistance, management model)


Pinho & Tavares (2009), Tracking Features in Image Sequences with Kalman Filtering, Global Optimization, Mahalanobis Distance and a Management Model, Computer Modeling in Engineering & Sciences 46(1):51-75

Pinho & Tavares (2009), Comparison between Kalman and Unscented Kalman Filters in Tracking Applications of Computational Vision, VipIMAGE 2009


Motion Tracking • Kalman Filter

– Optimal recursive Bayesian stochastic method– One of its drawbacks is the restrictive assumption of Gaussian

posterior density functions at every time step• Many tracking problems involve non-linear motions (i.e. human gait)

Image Processing and Analysis: Applications and Trends 34João Manuel R. S. Tavares

Motion Tracking• Example: tracking marks in gait analysis (Kalman filter,

Mahalanobis distance, optimization, management model)


Prediction Uncertainty Area Measurement Correspondence Result

Pinho et al. (2005), Human Movement Tracking and Analysis with Kalman Filtering and Global Optimization Techniques, ICCB 2005, 915-926Pinho & Tavares (2009), Tracking Features in Image Sequences with Kalman Filtering, Global Optimization, Mahalanobis Distance and a Management Model, Computer Modeling in Engineering & Sciences 46(1):51-75

(5 frames)



Pinho et al. (2007), Efficient Approximation of the Mahalanobis Distance for Tracking with the Kalman Filter, International Journal of Simulation Modelling 6(2):84-92

(547 frames)

Pinho et al. (2005), A Movement Tracking Management Model with Kalman Filtering, Global Optimization Techniques and Mahalanobis Distance, LSCCS, Vol. 4A:463-466

Motion Tracking• Example: tracking mice in long image sequences (Kalman

filter, Mahalanobis distance, optimization, management model)


Motion Tracking• Unscented Kalman Filter

– A set of sigma-points from the distribution of the state vector is propagated through the true non-linearity, and the parameters of the Gaussian approximation are then re-estimated

– Addresses the main shortcomings of Kalman Filter, as well as of Extended Kalman Filter, and is more suitable for non-linear motions


Motion Tracking• Example: tracking the centre of a square that is moving

according to a non-linear model (Kalman Filter (KF) and Unscented Kalman Filter (UKF))


Motion equations:

0 0

22( 1) 12.51 ,12.51

with 6, 10

x x ii iy xi i

x y

= + − +−= + −

= =

#6

+ predictionsx measurementsx corrections

Kalman Filter results:


(8 frames)

Motion Tracking• Example: tracking the centre of a square that is moving

according to a non-linear model (Kalman Filter (KF) and Unscented Kalman Filter (UKF)) – cont.


Motion equations:

0 0

22( 1) 12.51 ,12.51

with 6, 10

x x ii iy xi i

x y

= + − +−= + −

= =

(8 frames)

Tracking error (predicted/real state)


Motion Tracking• Example: tracking the motion of three mice in a real image

sequence (Kalman Filter (KF) and Unscented Kalman Filter (UKF))

+ predictionsx measurementsx corrections

(22 frames)




sequence (Kalman Filter (KF) and Unscented Kalman Filter (UKF)) –cont.

Kalman Filter results Unscented Kalman Filter results


(22 frames)



sequence (Kalman Filter (KF) and Unscented Kalman Filter (UKF)) –cont.


(22 frames)Tracking error (predicted/real state)

Motion Tracking• Influence of the adopted filter: Kalman Filter (KF) and

Unscented Kalman Filter (UKF)– If the motion is highly nonlinear, then UKF justifies its superior

computational load– Otherwise, KF with the undertaken matching (association)

methodology can accomplish efficiently the tracking– Hence, the decision between KF or UKF is application-dependent

• Frequently, UKF gets superior results• However, when the computational load is somewhat constrained, KF

with a suitable matching strategy can be a good tracking solution


Analysis of Objects:Matching, Registration and

Morphing

Analysis of Objects• Matching

– It is regularly used in the computational analysis of objects from images, for example, to register (i.e. align) objects, recognize objects, attain 3D information, analyze the motion tracked, and so forth

– Generally, it is achieved by considering invariant objects’ characteristics, as curvature, or displacements in a global space (like in modal space)

– Common problems: occlusion, non-rigid deformations, high shape variations, etc.


Analysis of Objects• Registration

– It is commonly required in order to compare objects in images acquired at different time instants or according to distinct conditions

– It is essential, for example, in Medicine to follow up the evaluation of patients’ diseases from images

– Usually, it is achieved by considering objects’ characteristic features, as points of maximum curvature, and their matching followed by the estimation of the involved transformation

– Common problems: key and invariant features not easily identified, occlusion, non-rigid deformations, severe shape variations, etc.


Analysis of Objects• Morphing (i.e. simulation)

– It is an especially used in Computer Graphics, but also very useful in the analysis of objects from images, for example, to estimate the deformation involved between two objects or between two configurations of an object, to simulate the transitions between two shapes acquired under a high temporal gap, etc.

– Normally, it is attained by considering simple geometric transformations

– However, when it must be considered the real behavior of the objects, physical methodologies and modeling as, for example, FEM, should be considered

• Common difficulties are related with the estimation of the involved forces and with the properties of the adopted (virtual) material

• The adequate matching of the objects is crucial


Matching• Using physical or geometrical modeling and modal matching


Modeling(physical or geometrical)

Eigenvalues / eigenvectors computation

Matching matrix

assembly

Contour 1

Contour 2

Matches achievement(optimization)

Modeling(physical or geometrical)

Eigenvalues / eigenvectors computation

Bastos & Tavares (2006), Matching of Objects Nodal Points Improvement using Optimization, Inverse Problems in Science and Engineering 14(5):529-541


• Example: matching contours in dynamic pedobarography (FEM, modal matching, optimization)

Matching


Original images Matched contours

Bastos & Tavares (2004), Improvement of Modal Matching Image Objects in Dynamic Pedobarography using Optimization Techniques, Lecture Notes in Computer Science 3179:39-50

camera mirror

contact layer + glass

reflected light glass

pressure opaque layer

lamp

lamp transparent layer

Tavares & Bastos (2010), Improvement of Modal Matching Image Objects in Dynamic Pedobarography using Optimization Techniques, Progress in Computer Vision and Image Analysis, Chapter 19, 339-368


Matching• Example: matching contours and surfaces in dynamic

pedobarography (FEM, modal analysis, optimization)


Image of dynamic pedobarography

Tavares & Bastos (2005), Improvement of Modal Matching Image Objects in Dynamic Pedobarography using Optimization Techniques, Electronic Letters on Computer Vision and Image Analysis 5(3):1-20

Matching foundbetween two contours

Matching found between two intensity (pressure) surfaces (2 views)

Matching found between iso-contours (2 views)


Registration• Registration of contours in images (geometrical modeling,

optimization, dynamic programming)


Oliveira & Tavares (2008), Algorithm of dynamic programming for optimization of the global matching between two contours defined by ordered points, Computer Modeling in Engineering & Sciences 31(11):1-11


Registration• Example: registration of contours in images (geometrical

modeling, optimization, dynamic programming)


Original images and contours

Matched contours before registration

Matched contours after registration

Oliveira & Tavares (2009), Matching Contours in Images through the use of Curvature, Distance to Centroidand Global Optimization with Order-Preserving Constraint, Computer Modeling in Engineering & Sciences 43(1):91-110


Registration• Example: registration of images in pedobarography

(geometrical modeling, optimization, dynamic programming)


Original images and contours Contours and images before and after registration

Oliveira et al. (2009), Rapid pedobarographic image registration based on contour curvature and optimization, Journal of Biomechanics 42(15):2620-2623João Manuel R. S. Tavares

Registration• Example: registration of images in pedobarography (Fourier

transform)


Original images Images before andafter registration

Oliveira et al. 2010, Registration of pedobarographic image data in the frequency domain, Computer Methods in Biomechanics and Biomedical Engineering (in press)


Registration• Example: registration of images in pedobarography (Hybrid

method: Contours registration or Fourier transform based registration + Optimization of a Similarity Measure – MSE, MI or XOR)


Original images Images before andafter registration

Oliveira & Tavares 2010, Novel Framework for Registration of Pedobarographic Image Data, Medical & Biological Engineering & Computing (submitted)


Registration• Registration of image sequences in dynamic

pedobarography (spatial and temporal registration)


Oliveira & Tavares 2010, Spatio-temporal Registration of Pedobarographic Image Sequences, Journal of Biomechanics (submitted)

Registration• Example: registration of image sequences in dynamic

pedobarography (spatial and temporal registration)

Image Processing and Analysis: Applications and Trends

57


Original sequencesbefore registration

Preprocessed sequences

Original image sequences

Sequencesafter registration

57

Morphing


• Physical morphing/simulation of contours in images (FEM, modal analysis, optimization, Lagrange equation)


• Example: morphing contours in images (FEM, modal analysis, optimization, Lagrange equation)

Matching found

Deformations estimated

Morphing


Gonçalves et al. (2008), Segmentation and Simulation of Objects Represented in Images using Physical Principles, Computer Modeling in Engineering & Sciences 32(1):45-55

Original images


3D Reconstruction

3D Reconstruction• It is intended to achieve the 3D reconstruction of objects or

scenes from images• In this area, the following methodologies are common:

external shapes – active techniques (with energy projection or relative motion), passive techniques (without energy projection nor relative motion) and of space carving; inner shapes – 2D segmentation (contours, for example) and data interpolation

• Usually, it involves tasks of camera calibration, data segmentation, matching, triangulation and interpolation

• Common problems: geometric distortions, bad or unstable illumination, occlusion, noise, multiple objects, complex shapes and topologies, etc.


3D Reconstruction• Example: 3D reconstruction of organs from medical images

(segmentation 2D, marching cubes, Delaunay)


Segmentation done in a 2D slice and the 3D reconstruction

obtained

3D reconstruction of some structures of an arm

Perdigão et al. (2005), Sobre a Geração de Malhas Tridimensionais para fins Computacionais a partir de Imagens Médicas, CMNI 2005

3D Reconstruction• Example: 3D reconstruction of organs from medical images

(segmentation 2D, loft, smooth, Delaunay)


Segmentation donein a 2D slice

Pelvic floorreconstructed

Reconstructed organs from a pelvic

cavityPimenta et al. (2006), Reconstruction of 3D Models from Medical Images: Application to Female Pelvic Organs, CompIMAGE 2006, 343-348Alexandre et al. (2007), 3D reconstruction of pelvic floor for numerical simulation purpose, VipIMAGE2007, 359-362

slices

3D Reconstruction• Example: 3D reconstruction of a scene using a technique

of active vision (dense stereo vision)


Disparity mapobtained

Original image pair

Azevedo et al. (2006), Development of a Computer Platform for Object 3D Reconstruction using Active Vision Techniques, VISAPP 2006, 383-388

3D Reconstruction• Example: 3D reconstruction of objects by space carving


Azevedo et al. (2008), 3D Object Reconstruction from Uncalibrated Images using an Off-the-Shelf Camera, Advances in Computational Vision and Medical Image Processing: Methods and Applications, 117-136

Original images Computational 3D model built voxelizedand poligonized

3D Reconstruction• Example: 3D reconstruction of objects by space carving


Original images Computational 3D model built voxelizedand poligonized

Azevedo et al. (2010), Three-dimensional reconstruction and characterization of human external shapes from two-dimensional images using volumetric methods, Computer Methods in Biomechanics and Biomedical Engineering 13(3): 359-369

Conclusions

Conclusions

• The area of image processing and analysis is very complex and demand, but of raised importance in many domains

• Numerous hard challenges exist, as for example, adverse conditions in the image acquisition process, occlusion, objects with complicate shapes, with topological variations or undergoing complex motions

• Considerable work has already been developed, but important and complex goals still to be reached

• Methods and methodologies of other research areas, as of Mathematics, Computational Mechanics, Medicine and Biology, can contribute significantly for their reaching

• For that, collaborations are welcome


Research Team(Computational Vision)

Research Team (Computational Vision)

• PhD students (15):– In course: Raquel Pinho, Patrícia Gonçalves, Maria

Vasconcelos, Ilda Reis, Teresa Azevedo, Daniel Moura, Zhen Ma, Elza Chagas, Victor Albuquerque, Francisco Oliveira, Eduardo Ribeiro, António Gomes, João Nunes, Alex Araujo, Sandra Rua

• MSc students (13):– In course: Carlos Faria, Elisa Barroso, Ana Jesus, Veronica

Marques, Diogo Faria– Finished: Daniela Sousa, Francisco Oliveira, Teresa Azevedo,

Maria Vasconcelos, Raquel Pinho, Luísa Bastos, CândidaCoelho, Jorge Gonçalves

• BSc students (2)– Finished: Ricardo Ferreira, Soraia Pimenta


Image Processing and Analysis:Applications and Trends


[email protected] www.fe.up.pt/~tavares

Slide Number 1OutlineIntroductionIntroductionIntroductionIntroduction: Usual Computational Pipeline for Image Processing and Analysis�SegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationSegmentationMotion TrackingMotion TrackingMotion TrackingMotion Tracking Motion TrackingMotion TrackingMotion TrackingMotion TrackingMotion TrackingMotion TrackingMotion TrackingMotion TrackingMotion TrackingAnalysis of Objects:�Matching, Registration and MorphingAnalysis of ObjectsAnalysis of ObjectsAnalysis of ObjectsMatchingMatchingMatchingRegistrationRegistrationRegistrationRegistrationRegistrationRegistrationRegistrationMorphingMorphing3D Reconstruction3D Reconstruction3D Reconstruction3D Reconstruction3D Reconstruction3D Reconstruction3D ReconstructionConclusionsConclusionsResearch Team�(Computational Vision)Research Team (Computational Vision)Slide Number 71

Image Processing and Analysis: Applications and …tavares/downloads/publications/...Image...

Documents

Transcript of Image Processing and Analysis: Applications and …tavares/downloads/publications/...Image...