Perceptual Organization of Curvilinear Structures Laurent Alquier Research Director : Chabane...

Perceptual Organization of Perceptual Organization of Curvilinear StructuresCurvilinear Structures

Laurent AlquierLaurent Alquier

Research Director : Research Director : Chabane OussalahProfessor

Thesis Advisor :Thesis Advisor :Philippe MontesinosAssistant Professor

- UNIVERSITE MONTPELLIER II -- Laboratoire de Génie Informatique et d 'Ingénierie de

Production, Nîmes -- September, 30th 1998 -

1 September 1999Laurent ALQUIER -

Image AnalysisImage Analysis

Intensity Image

Contours Images

Detection of visual cues

Construction of a symbolic representation


Visual PerceptionVisual Perception

Theories of Visual PerceptionTheories of Visual Perception Continuous flow of visual informationContinuous flow of visual information

Necessity to guide perceptionNecessity to guide perception


Perceptual OrganizationPerceptual Organization PrinciplesPrinciples

Pre-attentive phenomenon of groupingPre-attentive phenomenon of grouping Global visual properties : Global visual properties : SaliencySaliency Imposed to perception Imposed to perception beforebefore interpretation interpretation Elementary groupsElementary groups

Principle of "good shape"Principle of "good shape" Simplicity, closure, familiarity : Simplicity, closure, familiarity : StabilityStability

Useful properties Useful properties Generic, robust - Qualitative organization Generic, robust - Qualitative organization

Gestalt Theory : Wertheimer, Koffka - 1923Gestalt Theory : Wertheimer, Koffka - 1923

References


OverviewOverview

Introduction.Introduction.

Context and Objectives.Context and Objectives.

Organization using Saliency Networks.Organization using Saliency Networks.

High Levels of Organization.High Levels of Organization.

Contributions and Perspectives.Contributions and Perspectives.


ObjectivesObjectives Complete system of image analysisComplete system of image analysis

Segmentation - hypotheses - interpretationSegmentation - hypotheses - interpretation Application of psycho-vision principlesApplication of psycho-vision principles

Analysis of complex scenesAnalysis of complex scenes Restriction to shapes from contoursRestriction to shapes from contours Validation with artificial scenesValidation with artificial scenes Application to images of real scenesApplication to images of real scenes

Goal :Goal : Extract a set of elements of representationExtract a set of elements of representation Remain open to possible needs from future Remain open to possible needs from future

applicationsapplications


Overview of the systemOverview of the system

Edge detectionEdge detection

Three levels of organizationThree levels of organization Selection of salient structuresSelection of salient structures Extraction of elementary hypothesesExtraction of elementary hypotheses Organization into complex hypothesesOrganization into complex hypotheses

Hierarchic relationship between hypothesesHierarchic relationship between hypotheses

Application Application Detection of junctionsDetection of junctions Structural matchingStructural matching


OverviewOverview





Contributions and Perspectives.Contributions and Perspectives.


Detection of Curvilinear StructuresDetection of Curvilinear Structures Goal :Goal :

Select the most regular contoursSelect the most regular contours Complete discontinuitiesComplete discontinuities Extract stable structures ( noise, scale )Extract stable structures ( noise, scale )


Grouping by SaliencyGrouping by Saliency

PrinciplesPrinciples Estimate the visual quality of a structureEstimate the visual quality of a structure optimise this quality functionoptimise this quality function

Direct ApproachDirect Approach Extension Fields (Guy and Medioni, 1996)Extension Fields (Guy and Medioni, 1996) Stochastic Completion Fields (Williams and Jacobs, 1994)Stochastic Completion Fields (Williams and Jacobs, 1994)

Approaches by optimisationApproaches by optimisation Neural Networks (F. Mangin, 1994)Neural Networks (F. Mangin, 1994) Mean Field Annealing (L. Hérault, 1991)Mean Field Annealing (L. Hérault, 1991) Saliency Network (Shashua et Ullman, 1989) Saliency Network (Shashua et Ullman, 1989)


Saliency Network Saliency Network


Saliency Network (2)Saliency Network (2)

Selection of a PrimitiveSelection of a Primitive Definition of visual properties.Definition of visual properties.

Definition of Local NeighborhoodDefinition of Local Neighborhood

Network of locally connected elements.Network of locally connected elements.

Quality function for a groupQuality function for a group Estimate compatibility between elements of a group.Estimate compatibility between elements of a group. "Extensible" functions."Extensible" functions.

Definition of SaliencyDefinition of Saliency Quality of the best possible group for a given primitiveQuality of the best possible group for a given primitive


Local NeighborhoodLocal Neighborhood

Relationships between primitives.Relationships between primitives. Primitives linked using Primitives linked using Elements of Elements of

Connection.Connection. Properties of Properties of ProximityProximity and and CompatibilityCompatibility.. Method suited for Method suited for curved groupscurved groups..

Importance of neighborhood.Importance of neighborhood. Initialize optimisation of the network.Initialize optimisation of the network. Sets complexity of optimisation.Sets complexity of optimisation. Influence visual quality of groups.Influence visual quality of groups.


Quality FunctionQuality Function

Properties of ContinuityProperties of Continuity Proximity, smoothness, similarity.Proximity, smoothness, similarity.

FormalismFormalism Linear combination of opposed constraints.Linear combination of opposed constraints. "Internal" relationships."Internal" relationships.

Visual properties of groups expected.Visual properties of groups expected. "External" relationships."External" relationships.

Imposed by contours from image.Imposed by contours from image.

external internal

)F(j

jjk

kk RR


Structural SaliencyStructural Saliency

Recursive expressionRecursive expressionF is supposed to be an "extensible function"

Formal definition.Formal definition. Best sum of contributions around a primitive Best sum of contributions around a primitive

according to two directions.according to two directions.

)()(Max )( vv)( )v (v,

FFPSPV 2

)),,(Max,(Max),,(Max 21)(

11)(

1)( 2

111

NNee

NNe

eeeeeNN


Structural Saliency (2)Structural Saliency (2)

Example of recursive expressionExample of recursive expressionLocal term.

Contribution of neighbors.

Iterative optimisation.Iterative optimisation. Research of best neighbors.Research of best neighbors. Update of contributionsUpdate of contributions

)(),(

)(),(

)(

)(3

13

2

221

eRevP

eRevP

vQ

vRnkk

nkk

nk

nk



PropertiesProperties Local measurements - Global optimisation.Local measurements - Global optimisation. Completion of discontinuities.Completion of discontinuities. Saliency Map.Saliency Map. Grouping possible by following connections.Grouping possible by following connections.

Only one optimal group for each primitive.Only one optimal group for each primitive.



ApplicationApplication Organization of Pixels.Organization of Pixels.

Static neighborhood.Static neighborhood. Heavy computation.Heavy computation. Slow optimisation.Slow optimisation.

Organization of Chains.Organization of Chains. Dynamic Neighborhood.Dynamic Neighborhood. Reduced complexity.Reduced complexity. Fast optimisation. Fast optimisation.


Consequence of groupingConsequence of grouping

Normal edge linkingNormal edge linking

Organization of chainsOrganization of chains


Detection of Salient GroupsDetection of Salient Groups


Elementary GroupsElementary Groups

Classes of groupsClasses of groups Starting points along a global structure.Starting points along a global structure. "Attraction" phenomenon."Attraction" phenomenon.

Evaluation of groups Evaluation of groups Local saliency.Local saliency. Global saliency ( sum of saliency of primitives )Global saliency ( sum of saliency of primitives ) Accumulation of votesAccumulation of votes

SelectionSelection Threshold from evaluations.Threshold from evaluations.


Elementary Groups (2)Elementary Groups (2)


OverviewOverview




High Levels of Organization.High Levels of Organization. Primary Hypotheses.Primary Hypotheses. Complex groups and applications.Complex groups and applications.

Contributions and perspectives.Contributions and perspectives.


Primary Hypotheses Primary Hypotheses

Structural HypothesesStructural Hypotheses

Straight parts : Segments.Straight parts : Segments. Curved parts : Arcs.Curved parts : Arcs. Special points : Junctions, inflection points, corners.Special points : Junctions, inflection points, corners.

Principles of extractionPrinciples of extraction Detection from each elementary groupDetection from each elementary group

Scale, sensitiveness.Scale, sensitiveness. Fusion of primary hypotheses.Fusion of primary hypotheses.

Similarity - elimination of duplicates.Similarity - elimination of duplicates.


Primary Hypotheses (2)Primary Hypotheses (2)

Elements of symbolic representationElements of symbolic representation

Set of HypothesesSet of Hypotheses Role of ambiguities and errors.Role of ambiguities and errors. Certain amount of duplicates tolerated.Certain amount of duplicates tolerated.

Problems to solveProblems to solve DiscretizationDiscretization Structures with different scales within the same scene.Structures with different scales within the same scene. Output:Output:

Similar groups superimposedSimilar groups superimposed Exceptions to properties of continuity (occlusions and Exceptions to properties of continuity (occlusions and

junctions)junctions)


Hypotheses - Segments Hypotheses - Segments

Salient GroupsSalient GroupsSalient GroupsSalient Groups

Points of Points of interestinterest

Points of Points of interestinterest

Recursive division Recursive division (tolerance Es)(tolerance Es)

Recursive division Recursive division (tolerance Es)(tolerance Es)

Fusion of similar segments Fusion of similar segments (length,orientation)(length,orientation)

Fusion of similar segments Fusion of similar segments (length,orientation)(length,orientation)

Grouped segmentsGrouped segmentsGrouped segmentsGrouped segments

Elementary segments


Segments ( before organization )Segments ( before organization )


Segments ( after organization )Segments ( after organization )


Hypotheses - ArcsHypotheses - Arcs

Salient GroupsSalient GroupsSalient GroupsSalient Groups

Points of Points of InterestInterest

Points of Points of InterestInterest

Division according to Extremes of Division according to Extremes of Curvature Curvature

(Scale Ea)(Scale Ea)

Division according to Extremes of Division according to Extremes of Curvature Curvature

(Scale Ea)(Scale Ea)

Fusion of similar arcs Fusion of similar arcs (classe,superposition)(classe,superposition)

Fusion of similar arcs Fusion of similar arcs (classe,superposition)(classe,superposition)

Grouped ArcsGrouped ArcsGrouped ArcsGrouped Arcs

Classes of Elementary Arcs


Arcs ( after organization )Arcs ( after organization )


Points of InterestPoints of Interest


OverviewOverview


Context and Objectives. Context and Objectives.


High Levels of Organization.High Levels of Organization. Primary Hypotheses.Primary Hypotheses. Complex groups and applications.Complex groups and applications.



Complex groupsComplex groups

Example of applicationExample of application Extraction of multiple junctionsExtraction of multiple junctions Structural matching of junctionsStructural matching of junctions

MotivationsMotivations

Rich structural informationRich structural information Location of center, orientation of branchesLocation of center, orientation of branches Robust matching possibleRobust matching possible Precise location possible after matchingPrecise location possible after matching

Structures difficult to extractStructures difficult to extract Alterations from contour detectionAlterations from contour detection

Few publications in that areaFew publications in that area


Organization of junctionsOrganization of junctions

SegmentsSegmentsSegmentsSegments PointsPointsPointsPoints

Detection of Double JunctionsDetection of Double JunctionsDetection of Double JunctionsDetection of Double Junctions

Fusion of Similar JunctionsFusion of Similar Junctions(proximity of centers, similar branches)(proximity of centers, similar branches)Fusion of Similar JunctionsFusion of Similar Junctions(proximity of centers, similar branches)(proximity of centers, similar branches)

Multiple JunctionsMultiple JunctionsMultiple JunctionsMultiple Junctions

Intersections

Confirmation of center


Detection of double junctionsDetection of double junctions


Fusion into multiple junctionsFusion into multiple junctions


Structural MatchingStructural Matching

PrinciplePrinciple

Consistent labeling of two sets of Consistent labeling of two sets of junctionsjunctions

RequiresRequires

Direct comparison of junctionsDirect comparison of junctions

Evaluate motion (transformation)Evaluate motion (transformation)

Between two imagesBetween two images

Within a single imageWithin a single image

Conditional comparisonConditional comparison


Structural Matching (2)Structural Matching (2)

Method in two stagesMethod in two stages Temporal MatchingTemporal Matching

Elimination of improbable matchesElimination of improbable matches Spatial MatchingSpatial Matching

Elimination of improbable groups Elimination of improbable groups

PropertiesProperties Matching as perceptual groupingMatching as perceptual grouping Tolerate important differences between imagesTolerate important differences between images Mutual reinforcement of two type of organizationMutual reinforcement of two type of organization


OverviewOverview







ContributionsContributions

Saliency NetworksSaliency Networks Generic FormalismGeneric Formalism

Generalized to Organization of ChainsGeneralized to Organization of Chains Configurable Quality FunctionConfigurable Quality Function More stable optimisationMore stable optimisation

Choice of the most important connections Choice of the most important connections Extraction of final groupsExtraction of final groups

Selection of the most salient structuresSelection of the most salient structures

Shashua and Ullman, 1991 - Alter and Basri, 1997Shashua and Ullman, 1991 - Alter and Basri, 1997

References


Contributions (2)Contributions (2)

Organization of Geometric HypothesesOrganization of Geometric Hypotheses SegmentsSegments ArcsArcs Points of InterestPoints of Interest

Modular approachModular approach Global strategy of organizationGlobal strategy of organization Specialized modulesSpecialized modules

Hypotheses defined for only one scaleHypotheses defined for only one scale

Application to numerous types of scenesApplication to numerous types of scenes

Mohan and Nevatia, 1992 - Sarkar and Boyer, 1993 - Mohan and Nevatia, 1992 - Sarkar and Boyer, 1993 - Gao and Wong, 1993Gao and Wong, 1993

References


Contributions (3)Contributions (3)

Perceptual Organization of JunctionsPerceptual Organization of Junctions Detection of Elementary JunctionsDetection of Elementary Junctions Organization into Multiple JunctionsOrganization into Multiple Junctions

Structural MatchingStructural Matching Cooperation between Spatial Perceptual Cooperation between Spatial Perceptual

Organization and Temporal Matching.Organization and Temporal Matching.

Matas and Kittler, 1993 - Chang and Aggarwal, 1997Matas and Kittler, 1993 - Chang and Aggarwal, 1997

References


ResultsResults

Stable results Stable results Generic parameters for classes of scenesGeneric parameters for classes of scenes Robust in case of perturbationsRobust in case of perturbations

Reasonable computation time on low-end systemsReasonable computation time on low-end systems ExampleExample

PC - Pentium 100 - 65 Mo RAMPC - Pentium 100 - 65 Mo RAM Image 800x600 pixelsImage 800x600 pixels 500 chains in a scene500 chains in a scene Grouping with Saliency Network : 30sGrouping with Saliency Network : 30s Global computation time : 5 minGlobal computation time : 5 min


Application : Outdoor ScenesApplication : Outdoor Scenes


Application : Satellite ImagingApplication : Satellite Imaging


Junction Matching Junction Matching


Junction Matching (2)Junction Matching (2)


Short term extensionsShort term extensions

Organization of Generic ObjectsOrganization of Generic Objects

Quantitative evaluation of resultsQuantitative evaluation of results Qualitative results only for nowQualitative results only for now Numerous parametersNumerous parameters Empirical definition for nowEmpirical definition for now

Automatic validation of hypothesesAutomatic validation of hypotheses

Multi-scale detectionMulti-scale detection

Hierarchical Structural Matching Hierarchical Structural Matching Top-down approachTop-down approach


PerspectivesPerspectives

Integration with other visual cuesIntegration with other visual cues Visual attention - trackingVisual attention - tracking

Laurent Iti, 1997 Caltech - Roch and Ullman, 1985Laurent Iti, 1997 Caltech - Roch and Ullman, 1985

References


Perspectives (2)Perspectives (2) Automatic Indexation of ModelsAutomatic Indexation of Models

Aspect graphs Aspect graphs

Pope and Lowe, 1996Pope and Lowe, 1996

References

Perceptual Organization of Curvilinear Structures Laurent Alquier Research Director : Chabane...

Documents

Transcript of Perceptual Organization of Curvilinear Structures Laurent Alquier Research Director : Chabane...