Hierarchical Finite-State Modeling for Texture ... · Segmentation with Application to Forest...

Hierarchical Finite-State Modeling for Texture

Segmentation with Application to Forest Classification

Giuseppe Scarpa, Michal Haindl, Josiane Zerubia

To cite this version:

Giuseppe Scarpa, Michal Haindl, Josiane Zerubia. Hierarchical Finite-State Modeling for Tex-ture Segmentation with Application to Forest Classification. [Research Report] RR-6066, IN-RIA. 2006, pp.47. <inria-00118420v2>

HAL Id: inria-00118420

https://hal.inria.fr/inria-00118420v2

Submitted on 18 Dec 2006

HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, estdestinee au depot et a la diffusion de documentsscientifiques de niveau recherche, publies ou non,emanant des etablissements d’enseignement et derecherche francais ou etrangers, des laboratoirespublics ou prives.

https://hal.archives-ouvertes.fr

https://hal.inria.fr/inria-00118420v2

a p p o r t

d e r e c h e r c h e

ISS

N02

49-6

399

ISR

NIN

RIA

/RR

--60

66--

FR

+E

NG

Thème COG

INSTITUT NATIONAL DE RECHERCHEEN INFORMATIQUE ET EN AUTOMATIQUE

Hierarchical Finite-StateModeling for TextureSegmentationwith Application to Forest

Classification

GiuseppeScarpa— Michal Haindl — JosianeZerubia

N° 6066

December2006

UnitéderechercheINRIA Sophia Antipolis2004,routedesLucioles,BP 93, 06902SophiaAntipolis Cedex (France)

Téléphone: +334 92 38 77 77— Télécopie: +334 92 38 77 65

Hierar chical Finite-StateModeling for TextureSegmentationwith Application to ForestClassification

GiuseppeScarpa∗, Michal Haindl† , JosianeZerubia

ThèmeCOG— SystèmescognitifsProjetsAriana

Rapportderecherchen° 6066— December2006— 47pages

Abstract: In this researchreportwe present a new modelfor texture representationwhich is par-ticularly well suitedfor imageanalysis andsegmentation. Any imageis first discretizedandthena hierarchical finite-stateregion-basedmodel is automaticallycoupledwith the databy meansofa sequential optimizationscheme, namely the Texture Fragmentation and Reconstruction (TFR)algorithm.TheTFR algorithmallows to modelboth intra- andinter-texture interactions,andeven-tually addressesthesegmentationtaskin a completelyunsupervisedmanner. Moreover, it providesa hierarchicaloutput,asthe usermay decidethe scaleat which the segmentationhasto be given.Testswerecarriedoutonbothnaturaltexturemosaicsprovidedby thePrague Texture SegmentationDatagenerator Benchmark andremote-sensingdataof forestareasprovidedby theFrenchNationalForestInventory(IFN).

Key-words: Texturesegmentation,classification,co-occurrencematrix,structuralmodels,Markovchain,texturesynthesis,forest classification.

Thiswork wascarriedoutduringthetenureof anERCIM fellowship(Scarpa’s postdoc).

∗ ERCIM Fellow hostedby UTIA/CRCIM Prague(CZ) andINRIA SophiaAntipolis (F).Now heis anAssistantProfessorof theUniversity“FedericoII” of Naples(I). Email: [email protected]

† PatternRecognitionDepartment,Instituteof InformationTheory andAutomationof theCzechAcademyof Sciences,Prague(CZ). Email: [email protected]

Modèlehiérarchiqueà étatsfinis pour la segmentationdetexture. Application à la classificationde forêts

Résumé: Danscerapport derecherche,nousproposonsun nouveaumodèlepourla représentationdestexturesqui estparticulièrementbienadaptéàl’analyseetàla segmentationdesimages. Chaqueimageestd’aborddiscrétisée.Ensuite,cettereprésentationdiscrèteestautomatiquementassociéeà un modèlehiérarchiqueà étatsfinis fondésur les régions,grâceà uneoptimisation séquentielle,via l’algorithmeTexture Fragmentation and Reconstruction (TFR).Le TFR permetla modélisationsoit desinteractionsintra-textures,soit desinteractionsentretexturesdifférenteset doncil résoutle problèmede la segmentationdemanièrecomplètementnonsupervisée.En outre, il fournit unesolutionhiérarchiquequi peutêtreinterprétéeàdifférenteséchellesspatialesenfonction desbesoinsdel’util isateur. Différentstestsdel’algorithmeont étéfaitssurdesimagestexturéesfourniesparlePrague Texture Segmentation Datagenerator Benchmark etsurdesimagesdetélédétectiondeforêtsfourniesparl’InventaireForestierNational(IFN).

Mots-clés: Segmentationdetexture,classification,matricesdeco-occurrence,modèlestructural,chaînedeMarkov, synthèsedetexture,classificationdesforêts.

Hierarchical Finite-State Texture Modeling 3

Contents

1 Intr oduction 6

2 TS-MRF model-basedsegmentation 7

3 TextureFragmentation and Reconstruction(TFR) algorithm 93.1 Hierarchicalfinite-statetexturemodeling . . . . . . . . . . . . . . . . . . . . . . . 93.2 Generaloptimizationscheme. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .123.3 Color-basedclustering(CBC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .153.4 Spatial-basedclustering (SBC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .163.5 Regionmerging: theregion gain . . . . . . . . . . . . . . . . . . . . . . . . . . . .173.6 Enhancedregiongain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

4 Experimental Results 204.1 ThePragueTextureSegmentationDatageneratorBenchmark . . . . . . . . . . . . . 20

4.1.1 Performanceassessment . . . . . . . . . . . . . . . . . . . . . . . . . . . .214.1.2 Comparativesegmentation algorithms . . . . . . . . . . . . . . . . . . . . . 244.1.3 Texturemosaicsegmentationresults . . . . . . . . . . . . . . . . . . . . . . 25

4.2 Applicationto remote-sensingimagescoveringforestareas. . . . . . . . . . . . . . 40

5 Conclusion 43

RR n° 6 0 6 6

4 Scarpa & Haindl & Zerubia

Glossary

Acronyms

AR3D 3-D Auto RegressivemodelCBC Color-BasedClustering

CBIR Content-BasedImageRetrievalEDISON EdgeDetectionandImageSegmentatiONsystem

EM Expectation-Maximization estimationalgorithmGM GaussianMixture

GMRF Gauss-MRFmodelH-RAG HierarchicalRegionAdjacency Graph

KLD Kullback-LeiblerDivergenceMDL Minimum DescriptionLengthvalidationcriterionMRF Markov RandomFieldPCA PrincipalComponentAnalysisRAG RegionAdjacency GraphSBC Spatial-BasedClusteringTFR TextureFragmentation andReconstructionalgorithm

TFR+ TFR with KLD-basedregiongainTP TransitionProbability

TPM TransitionProbabilityMatrixTS-MRF Tree-StructuredMarkov RandomField

INRIA


Symbols

\ setminus〈·〉 statisticalaverage

ǫi(·, ·) measureerrortolerantto refinementηj(s) neighborof s in directionj

Ω quantizedcolor spaceB numberof spectralbandsC commissionerror

CA weightedaverageclassaccuracyCC objectprecisionCI comparisonindexCO recallCS correctdetection

D(p‖q) Kullback-Leiblerdiscrimination betweenp andqEA meanclassaccuracy estimate

GCE globalconsistency errorGt split gain

Gi,GiKL gainof region i andits modifiedversionwith KLD, respectivelyG∗ regiongain thresholdI typeI error

II typeII errorK numberof clustersin thek-meansalgorithmL numberof statesat thefinestrepresentationlevel

LCE local consistency errorME missedMS mappingscore

N, N numberof texturesin theimageandits estimateNc numberof connectedsubregionsof thecolor-quantizedregion c

NE noiseO omissionerror

OS over-segmentationP,Q empiricaltransitionprobability matrixandits modifiedversion,respectivelyRi regioncorrespondingto theterminal statei

RM rootmeansquareproportionestimationerrorS imagesupportSc regioncorrespondingto thequantizationcolor cSc

n connectedsubregionof Sc

Sω−→j ω′ s ∈ Sω, ηj(s) = ω′T c

n transitionprobabilitymatrix associatedwith regionScn

US under-segmentation

RR n° 6 0 6 6


1 Intr oduction

Imagesegmentation[4, 7, 13,23] is a low-level processingwhich is of critical importancefor manyapplicationsin several domains,like medical imaging,remotesensing,sourcecoding,andso on.Although it hasbeenwidely studiedin the last decadesin many casesit remainsstill open,asfortexturedimages, wherethespatialinteractionsmaycover longrangesaskingfor highordercomplexmodeling.

Therearea large numberof approachesto segmentation, but for the sake of brevity, hereweconfineourselvesto reviewing only thosethathavebeentestedusingthesamebenchmarkingsystem[14] asweuse,andwhichthereforeserveaspointsof comparison.In [12] imageblocksaremodeledby meansof localGaussMarkov RandomFields(GMRF) andthesegmentationis performedin theparameterspaceby assumingan underlyingGaussianMixture. Similar to the previous,but withanauto-regressive 3-D model(AR3D) in placeof theGaussMRF, is themethodpresentedin [13].In [8] anapproach,namely theJSEG,is presentedwheresegmentationis achieved in two steps:acolor quantizationfollowedby a processingof the label mapwhich accountsfor spatial interaction.Anothermethodtakenin considerationis thesegmentationalgorithmunderlyingthecontent-basedimageretrieval system Blobworld [4]. HereaGaussianMixturemodelis assumedin afeaturespace,wherecontrast,anisotropy andpolarity are the salient texture descriptors,and the EM algorithmcarriesout the clustering. Finally, the algorithm presentedin [5] (EDISON) combinesa region-basedapproachwith a contour-basedone,hencebalancingtheglobalevidencewhich characterizesa region-basedmodelwith the local informationtypically dominantin thecontourmodeling.

In this researchreportwe presenta novel methodfor unsupervisedtexturesegmentation,wheretheimageto besegmentedis first discretizedandthenahierarchicalfinite-stateregion-basedmodelis automaticallycoupledwith the databy meansof a sequentialoptimizationscheme,namelytheTexture Fragmentation and Reconstruction (TFR) algorithm.Themodelallows to take into accountbothintra-andinter-textureinteractions,andeventuallyallows to addressthesegmentationtaskin acompletelyunsupervisedmanner. Moreover, it providesahierarchicaloutput,suchthattheusermaydecidethescaleatwhich thesegmentationhasto begivenon thebasisof thespecificapplication.

TheTFRis basicallycomposedof two steps.Theformerfocusesontheestimationof thestatesatthefinestlevel of thehierarchy, andisassociatedwith animagefragmentation,or over-segmentation.The latterdeals with thereconstructionof thehierarchy representingthetextural interactionat dif-ferentscales.Thereconstructionpartis controlledby a measurenamedregion gain which accountsfor thespatialscaleof thecorrespondingregion, andfor the“attraction” operatedby theneighbor-ing regionson it. In particulartwo differentgainshave beendefinedandcomparedexperimentingwith datageneratedby the benchmarksystem[14]. A comparisonwith othermethodsusing thesamebenchmarkwasconsideredaswell. Finally, wehaveconsideredaverycritical remotesensingapplication,which is the forestsegmentation[15], for which traditionalcolor-basedsegmentationmethodsusuallyfail, askingfor texture-basedalgorithms.Thedatain this casewerecurtesyof theFrenchNationalForestInventory (IFN).

INRIA


2 TS-MRF model-basedsegmentation

Thetexturesegmentationalgorithmwhich will bepresentedlater in this researchreport,makesuseof acolor-basedsegmentationasafirst step.Amongall thepossiblechoiceswehaveselectedfor thispurposethe tree-structured MRF (TS-MRF)model-basedalgorithmpresentedin [7, 23] andbrieflyrecalledhere.This algorithmhasseveralcharacteristicswhich areattractive in this context. It usesa MRF prior modeling which helpsto regularizeelementary regionsand improve the robustnesswith respectto the presenceof noise. Furthermore,the datalikelihooddescriptionis basedon amultivariateGaussianmodelingwhich takesinto accountthecorrelationin thecolor space.Finally,its treestructuredformulation,similar to that of the tree-structuredvector quantizationalgorithm[11], speedsup the processing,andensuresconvergenceto the desirednumberof clusters.In thefollowing abrief description of theTS-MRFalgorithmis given.

A randomfield X definedonalatticeS is saidto beaMRF with respectto agivenneighborhoodsystemif theMarkovian propertyholdsfor eachsites. Thedistribution of a positive MRF canbeprovedto have aGibbsform [10], thatis:

p(x|θ) =1

Zexp[−U(x, θ)], (1)

with U(x, θ) =∑

c∈C Vc(xc, θ), wherex is therealization of thefield X, θ is thesetof parametersof themodel,theVc functionsarecalledpotentials,U denotestheenergy, Z is anormalizingconstantthatdependson θ, andc indicatesa cliqueof the image.Note thateachpotentialVc dependsonlyon the valuestaken on the cliquesitesxc = xs, s ∈ c and, therefore,accounts only for localinteractions.As aconsequence,localdependenciesin X canbeeasilymodeledby definingsuitablepotentialsVc(·). In particularthe secondorderPottsMRF model [2] is considered in this work,whereonly pairwisecliquesareassociatedwith notnull potentials,thatis:

Vc(xc) =

β if xp 6= xq, p, q ∈ c0 otherwise

(2)

whereβ > 0 is themodelparameter.Dueto theinherenthigh complexity of this model,which canbeoptimizedby stochasticrelax-

ation algorithmsor othersimilar procedures,a faster algorithmfor unsupervisedimagesegmenta-tion, which is basedon “tree-structured”MRF modeling,hasbeendevelopedin [7].

Let usnow considera K-classimagesegmentationproblemandmodeltheunknown labelmapwith a randomfield X definedon the latticeS of the imagey to be segmented.Sucha problemcanbeviewedasanestingof severalsegmentationproblemsof reducedcomplexity. Givenabinarytreestructurewith K terminalleaves,eachassociatedwith oneof theK class/region to besingledout, therewill be K − 1 internalnodes,eachrepresentingthe merging region of the descendantleaves/regions.Equivalently, eachof suchlargerregion representstheirregular1 (exceptfor therootnode)supportdomainfor its children regions, which correspondeither to terminal classesor tomerging classes.Actually, in the TS-MRF unsupervisedalgorithmsucha structureandthe corre-spondingimagepartitioningarerecursively singledoutby top-down induction, whereeachstepisa

1Not rectangular.

RR n° 6 0 6 6


binaryregion split (associatedwith a nodesplit) basedon a local binaryMRF like thePottsmodelpresentedabove,with K = 2, anddefinedonaproper, irregular, latticeshapedby theancestorsplits.

Thegrowth of thetree,thatis thechoiceof thenodeto besplit from timeto timeis controlledbya testlocal to eachnode,namelythesplit gain, which accountsfor thelikelihoodof thehypothesisof split of thecorrespondingnodewith respectto thehypothesisof nonsplit of thesamenode,onthe basisof the observed local testsegmentation(see[7] for further details). Also, whenall suchtestson thecurrentterminalleavesindicateto not split, the treegrowth is stoppedandthenumberof classesis automatically givenby thenumberof leaves.

Indeed,sincewe will usetheTS-MRFaspartof a morecomplex tool for texturesegmentation,the clustervalidation problemhasnot to be solved by the TS-MRF itself which, eventually, willsimplyover-segmenttheimage.For this reasonwedefinehereadifferentsplit gain, to notbemeantashypothesistestbut only asindicatorof thetotal distortiondecreaseobtainedby fitting separatelythetwo regionsof a split with local models.Let uslabelt thenode/region to besplit andber andltheright andleft childrenrespectively, thereforethesplit gain is definedas:

Gt(xt) =

pr(yr)pl(y

l)

pt(yt), (3)

wherext is the testsegmentation2, yi is the imageportionassociatedwith nodei, andpi(·) is thebestmatchingGaussiandistribution for the datayi. Eventually, the TS-MRF algorithmusedherediffers form theversionpresentedin [7] in theway how thetreegrowth is controlled.In particularthesplit gain definedabove doesnot containanadditionalterm,namelytheprior probabilityp(xt)(see[7]), which accountsfor theshaperegularity of the childrenregionsbut is unnecessaryin thiscontext. Also thegrowth is stoppedwhenthe requirednumberof classesis reachedandnot whenall gainsareundera threshold.

2A regionsegmentationis first computedand,hence,on thebasisof thecorrespondingsplit gain is validatedor not.

INRIA


3 TextureFragmentation and Reconstruction(TFR) algorithm

Theproposedsegmentationmethod,hereafterreferredto asTexture Fragmentation and Reconstruc-tion (TFR) algorithm, is basedon a hierarchicalfinite-statemodelingof the imagetextures,andisoptimizedby meansof a split-and-merge procedure.The former (top-down) stepaimsat decom-posingthe imageto be segmented(then,alsoeachof its textures)in a sufficiently large numberof components,henceit over-segmentsthe image. The latter (bottom-up)stepaimsat associatingrecursively thetexturecomponentspreviouslyextractedsothateachdifferenttextureis properlyre-constructed.Theprocessprovidesusat theendwith ahierarchicalsegmentationmap,asstructuredtexturescanberevealedat differentscales.By structured texturewe meana spatialpatternwhoseinteractionrange,i.e. theorder, is not clearlyboundedbut canberegardedasthesuperpositionofsimplerbounded-rangeinteractions. An imagepatterncontainingboth macroandmicro texturalinteractionsis suchanexample.

In general,acomplex scenariocanbealsoregardedasasingle textureat thecoarsestscale,and,in a sense,thesameclustervalidationproblem3 doesnot make sense,i.e. it is anill-posedproblem,if thescaleis not fixedsomehow. As anexample,considerthe front of a building with anarrayofwindows. At afinestscaleonelikely candistinguishtheglasses, theframes of thewindows,andthewalls. Then,at a coarserscale,framesandglassescanbeconsideredasa uniquetexture(window),sincethey arestronglyrelatedspatially, while at the coarsestscalewindow andwalls, which alsorelateto eachotherbut with longerrangespatialinteractions,mergein thebuilding texture.

The ill-positioning of theclustervalidationproblemis very commonin many computervisionapplications,and,beingawareof its strict correlationwith thespatialscale,hereinwewill provideamethodwhich givesahierarchicalsegmentationratherthanasinglesegmentationwith anestimated(somewhat“unreliable”)numberof regions.By doingso,wegetascale-dependentinterpretation oftheimage,representedby asetof nestedsegmentationswhichcanbeassociatedwith atreestructurewhereeachof its pruningcorrespondsto apossiblesegmentation.

3.1 Hierar chical finite-state texturemodeling

In orderto achieve thegoaldiscussedabove, we resortto anunderlyinghierarchical, discrete andregion-based modelingof the textures. As a consequence, a first basicstepis the transformationfrom the continuousspaceof the imageR

B×|S|, with B spectralbandsandsupportlatticeS, toa discreteoneΩ|S|, whereΩ is a finite set. Sucha processcanbe eitherjust a color quantizationdirectlyappliedto theoriginal imageor, moregenerally, aclusteringin any featurespacewheretheimagecanbe projectedfirst. In both cases,the cardinality of Ω fixesa “resolution” of the model,thatin thefirst casecorrespondsto a color resolution.Indeed,asit will beclearerlater, sincethis isaregion-basedmethod,thehigherthecardinalityof Ω, thesmallerthesizeof thebasicimageregionelements,hence,themodelresolutionis alsomeantasaspatialone.

A naturalquestionis then if themodel resolutionhasany limit, i.e. if thecardinalityof Ω canbechosenarbitrarily large. Thoughintuitively theansweris negative, it needssomefurtherdetailsaboutthemodelin orderto bemotivated.However, regardlessof themotivations,onecoulddoubt

3Thatis, theidentificationof thenumberof texturesin theimage.

RR n° 6 0 6 6


(a)

(b)

(c)

Figure 1: Hierar chical RegionAdjacencyGraph (H-RAG) texture representation: textile pat-tern (a), and two levelsof the H-RAG (b)-(c).

aboutthe texture descriptioncapability of the modeldueto the initial discretizationof the image.Actually, while this couldbe aratherserious limit in a synthesisproblem,it is not thatcritical in ananalysisproblemlike thesegmentation,andespeciallyin anunsupervisedsettingwhererobustness,ratherthenprecision,is themostrelevantissue.

Let usfocusnow ontheregion-hierarchicalmodelingaspects.A verycommongraphrepresenta-tion of asetof regionswhichaccountsfor their relativespatialpositioning,is theRegion AdjacencyGraph (RAG), that is a non-orientedgraphwherenodesareassociatedwith regions,andany con-nectionbetweentwo nodesindicatestheadjacency of thetwo correspondingregions.As anexampleconsiderthetextureof Fig.1(a). It is immediateto figurea three-level quantizedimages,with codesblue, black and red. Also, consideraselementaryregionsall the connectedareaswith the samecode. For suchan imagepartitioningin regions,the correspondingRAG is depictedin (c), whereeachnoderepresentsaconnectedregionandits colorrecallstheassociatedcolor-code.Furthermore,notice that thegraphis periodicbecauseof theperiodicalbehavior of the texture. Likewise, in (b)is shown the RAG for a two-level discretizationof the sametexture wherethe black and the redregionsaremergedin a mixing class. More generally, given a setof nestedimagepartitions,wecanassociate with it a cascadeof RAGs,namelytheH-RAG (HierarchicalRAG), or theHARAG(HierarchicalAttributedRAG) if thenodesarealsofeatured[9].

Sucha modelinghave beenusedmostly for content-basedimageretrieval applications(CBIR)[9, 24], wheretheregionsarespatiallyrelatedwithout any stationarityandthegoal is to find a sub-graph(hence,an imagearea)which well fits with the structuraldefinition of the searchedobject.Instead,for texture modelingsomestationarityis expected,asshown in the simplecaseof Fig.1wherea strongperiodicity is well visible and representedby its H-RAG as well. Likewise, anaperiodictexturewouldhave an associatedH-RAG which reflectsits statisticalproperties.

A compact,andactuallyenhanced,representationof theH-RAG for a stationarytextureis pos-sibleby meansof ahierarchicalfinite-statemodeling,asclarifiedby Fig.2w.r.t. theabove example.Let usconsidertheeightmainspatialdirections(north,north-east,east,etc...)andfor eachof themconsiderthe intra- andinter-region transitionprobabilities(TPs). Hence,for eachpartitioning de-gree,we candefinea set of eight statediagrams. For example,in (a) is shown the statediagram

INRIA


0.7

0.9 0.8

0.1 0.3

0.2

0.95

0.95 0.95

0.05 0.02

0.03

0.05

0.9 0.92

0.1

0.08

(a) (b) (c)

Figure 2: Fini te-statemodeling associatedwith the texture of Fig.1. Two-statemodel: eastdir ection (a). Thr ee-statemodel: east(b) and south (c) dir ections.

correspondingthe theeastdirectionwhenthetextureis reducedto only two states,while in (b) and(c) areplotteddiagramsw.r.t. the three-statepartition,for theeastandsouthdirections,respectively.Thisrepresentationis enhancedsincetheH-RAG only revealstheadjacency, regardlessof its weightanddirectionality. ApproximatedTPsareindicatedon thegraphsjust to give an ideaof their rela-tionshipwith thevisualappearanceof thetexture.To bemoreprecise,in talkingaboutTPswehaveto specifythespatialstepsize. In thefollowing, we simply refer to thepixel size,sincewe assumeto moveonapixel-by-pixel basis accordingto a8-connectedneighborhoodsystem.

Theinterpretationof this graphicalrepresentationis ratherimmediate.First, notethattheintra-regionTPsaccountfor theshapeof thetexturecomponentsatany region-scale.Consider, for exam-ple, thebluepatchesthat regularly occurin the texturesample. Dueto their rectangularshape,theassociatedintra-region TP in theverticaldirection(c) is largerthenthehorizontalone(b). Further-more,theremaining,inter-region,TPsaccountfor thespatialcontext, that is, therelativeoccurrenceandpositioningof theneighboringregions.For thetransitionsbetweenstatesit is very commontorefer to the conditional probabilitiesgiven that the state changes,which arescaleinvariant in thisparticularcase.Finally, observe that oncetheTPs areknown at a given level in thehierarchy, thenthey areautomaticallyobtainedalsofor the above levels and,eventually, oneonly hasto estimatetheseattributesat thefinest level.

Fromthesegmentationpoint of view, it is necessaryto move from theglobalcharacterizationslike theH-RAG or thefinite-statemodelsto a local characterizationat theregion level. Accordingto themodelinggivenabove,thenaturalattributesto associatewith any regionareits colorcodeandits localTPs.

To be moreformal, let us introducea few notations.Given a texture imageY ∈ RB×|S|, let

Z ∈ Ω|S| be its finest-level discrete-stateapproximation,whereΩ is the setof all possiblestates.At first glance,it couldseemthat thesestatesarejust thequantizationcolors,which is actuallynotnecessarytruesincethesamecolor mayoccurin a textureaccordingto differentconfigurationsofTP, andthendifferentstatesaredefined.Hence,let Sω ⊆ S = s ∈ S, Zs = ω bethesetof imagepixels with stateω ∈ Ω, thenthe associated|Ω| × 8 transitionprobability matrix (TPM), Pω, is

RR n° 6 0 6 6


computedas

Pω(ω′, j) =# of pixelss ∈ Sω suchthatηj(s) ∈ Sω′

# of pixelss ∈ Sω

=

|Sω−→j ω′ |

|Sω|∀ω′ ∈ Ω, 1 ≤ j ≤ 8,

(4)whereηj(s), is theneighborof pixel s in position(direction)j. Eventually, it T indicatesthe treestructurewhich representsa region hierarchy, accordingto our modela texture is definedby thetriple (Ω,P, T ), with P = Pωω∈Ω.

Finally, a local characterizationis easilyderivedby dealingwith theelementsof thepartitionofeachSω in its connectedsubregionsSn

ω , n = 1, . . . , Nω. In fact,any suchregion hasanown TPMwhichdiffersfrom theglobal one,i.e.

Pnω(ω′, j) =

|Snω−→j ω′ |

|Snω |

∀ω′ ∈ Ω, 1 ≤ j ≤ 8, (5)

but is relatedto, ascanbeeasily shown, by theweightedaverage:

Pω =1

|Sω|

Nω∑

n=1

|Snω |P

nω. (6)

This is just becauseweareconsideringa union of regions.Likewise,oncethehierarchy T is given,sincethe intermediatestatesare obtainedby merging the terminals,similar linear combinationsallow usto derive theTPMsatany coarserlevel.

3.2 Generaloptimization scheme

In theprevioussectionwe exploiteda discreteandhierarchicaltexturemodeling,regardlessof anyparticularapplication.Here,we will focuson theunsupervisedsegmentationproblemandproposeanoptimizationalgorithmfor thiscase.A few commentshaveto begivenabouttheproblemsetting.Sincewe are assuminganunsupervisedcontext, thenwe do not know a priori how many andwhatkind of texturesmay be found in the imageto be segmented. Also, we cannoteven restrict thevariety of imagepatternsof interestto a narrow-domain,sincewe want to focusexplicitly on thebroad-domaincase,beingawarethateventuallyalternative andparticularizedoptimizationscanbederived,for sure,if any restrictioncanbeassumed.

As a consequence,itemsto be estimatedarethe number of texturesand,for each texture, theterminal4 states,with correspondingTPMs, andthe hierarchicalrelationshipsby meansof whichintermediatestateswith associatedattributesarederivedby combiningtheterminals.

Thedeterminationof thenumberof texturesof a given image,classicallyreferredto asclustervalidation problem, is strictly relatedto thatof finding the internalstructureof eachsingletexture,especiallywhenlongrangeinteractionsarepresentin thetextures.Theexamplein Fig.3clarifiesthisobservation. The image(a) is composed of two textureswhoserepresentative statesat thecoarsestlevel areindicatedby w andz. In aH-RAG representationof atexture,thecoarserlevel corresponds

4Correspondingto thefinestlevel in thehierarchalmodeling.

INRIA


(a)

@@@

(b)w z

@@@

@

@

(c)

w

x y

@@@

@

@

@

@

(d)

w

u v

y

HHHHH

SSS

(e)

w u v y

@@@

@

@

(f)

w

u v y

Figure 3: Image structur e ambiguity. A texture mosaic (a) and several binary (b)-(d) andnon-binary (e)-(f) hierarchical modelings.

to the root of the hierarchicalstructureand, eventually, is associated with the texture as whole.Moreover, if animagecontainsmorethanonetexture,thanthewhole imagecanbeconsidereditselfasa texture whosehierarchicalstructureis simply obtainedby relatingthe marginal substructuresuntil a rootnodeis reached.In thissense,thestructuresshown in Fig.3(limited in depthfor sakeofsimplicity) represent bothintra-andinter-texturedependencies.A first observationto make is aboutthe ill-positioning of the clustervalidationproblem. Indeed,for suchan imagea humanobservercouldguessthat the texturesareactuallyfour andnot just two, andhencewe canexpectthat for acomputerthesedataareevenmoreconfusing.In termsof hierarchicalrepresentation,at thetoplevelthetwo differentcasescorrespondto (b) and(e).

Eventually, this ill-posedproblemneedsto beregardedfrom adifferentpointof view in ordertobereasonablysolved. So,ratherthanreally finding theexactnumberof textureswe will provide ahierarchical segmentation, thatis asetof nestedsegmentations(hence,coupledwith atreestructure)wherea singlesegmentationcorrespondsto a pruningof thefull tree.By doingso,asfinal productwe only have to provide the finest segmentationand the region hierarchy tree which univocallydeterminesany coarsersegmentation,leaving to theuserthechoiceof thepruning (i.e. thenumberof classes)whichbetterfits with thesubsequentapplication.

Indeed,this is ratherreasonablesincein mostof theapplications,segmentationis just anearlyprocessingwhich precedessomeotherones.As anexample,if we considerdatacompression andusea region-basedtechnique, thensegmentation is requiredasa first stepandthensingleregionsextractedareoptimally coded.If we referto theimageof Fig.3 (a), thenwe mayreasonablyexpectthe4-classpartition to bepreferableto the2-classone. In termsof hierarchicalrepresentationfortheprovidedsegmentation,thenwemayexpectastructurewhich figuresat thetoplevelslike(f). Asit canbeseen,thefull structure(f) correspondsto thedesired4-classsegmentation, while apruningof it would provide uswith a structurelike (b) which givesthe2-classpartition. Notice thata flatstructurelike (e)wouldnotcontainany substructurewhichcorrespondsto the2-classsegmentation.

Finally, let uslimit ourattentionto thecaseof “binary” structures.Comparing(d) and(f), wecanseeat first glancethat(d) is an approximationof (f) whenthebinary constraintis assumed.Indeed,from our point of view (d) is reacherthan(f), sinceit containsthesamesegmentationsas(f), plus

RR n° 6 0 6 6


?

CBC

?

SBC

?Regionmerging

?

?

6

?

6

?

6

?

6

pixel-level processing

region-level processing(negligible comp.load)

imagefragmentation(over-segmentation)

hierarchicaltexturereconstruction

Figure 4: Flow chart of the Texture Fragmentation and Reconstruction (TFR) algorithm.(CBC: color-basedclustering; SBC: spatial-basedclustering)

onemore,a 3-classsegmentation.This is simply dueto thepresenceof onemoreinternalnodeinthestructurewhich increases thenumberof possible prunings,i.e. imageinterpretations.Moreover,thebinaryconstraintsimplifiesthesearchof thetreestructureto beassociatedwith thesegmentationbecauseit limits thenumberof possiblestructuresto investigate.Fromtheabove considerations,inthefollowing weonly dealwith binaryhierarchies.

Let us turn now back to the optimizationproblem. As we re-formulatethe clustervalidationproblem,it is no more a critical part of the optimization. Therefore,it is partially solved whenthe global H-RAG is determined,andcanbe completedby any application-dependent criterion tobe applied on the final hierarchical segmentationprovided which, actually, hasjust to identify anoptimalpruning.

Therefore,we canneglect thenumberof textures,thenwhat remainsto beestimatedis a “rea-sonable”number5 of terminalstatesandtheglobalhierarchicaltree. Theoptimizationschemewepropose(seeFig.4)is quitesimplebut effective,asit will beshown by theexperimental tests.Noticefirst that theextractionof thestatesandthehierarchy areperformedin two separateandsequentialsteps.Theformer, composedby theblocksCBC (Color-BasedClustering) andSBC(Spatial-BasedClustering),dealing with theestimateof thestates,the latter focusedon thehierarchy. Thesplit oftheformerin two partsis justifiedby thefactthata statein our modelis characterizedby meansof

5Theterm“reasonable”remindsthatby choosingthatnumberwearefixing somehow theresolutionof thealgorithm.

INRIA


theregion response,eitherin theoriginal color spaceor in a transformedone,andtheTPM whichaccountsfor shapeandneighborhoodof the regions. Furthermore,while the color responseis apixel-wisefeature,theTPM is a region-wisecharacteristic.For thesereasonsweproposeto processcolorandspatialinformation independentlyin asequentialway. TheCBCblockwill createacertainnumberof partial-states,namelythecolor-states,discriminatedjust by thecolor, andthentheSBCwill furthersplit them onthebasisof theregion-basedfeatures.SinceSBC,aswell asthesubsequentregionmerging,worksat region-level (it handlestheelementaryconnectedregionscreatedby CBC)the associated computationalload is negligible w.r.t. the CBC burden. Region merging, or statemerging, is nothing but a sequentialbinarycombinationof thestates drivenby a specificparameterwhichaccountsfor themutual spatialrelationshipsamongthestates.

Let usdetailnow thesinglestepsof theTFR algorithm.

3.3 Color-basedclustering (CBC)

Thissegmentationstepis critical in thesensethatcolor informationis handledonly hereandregard-lessof spatialinteraction.Sinceonehasto extracta certainnumberof color-states,whathasto beperformedis acolor-basedsegmentationor clustering.Hence,CBCcanbeapproachedby usingoneof themany well known methodsfor this kind of problem,like vectorquantization[11], low orderMRF-basedalgorithms[10, 17],andsoon.

As CBC we have usedthe tree-structuredMRF (TS-MRF) algorithm[7]. This algorithmhasseveral featureswhich areattractive in this context. It usesa MRF prior modelingwhich helpstoregularizeelementaryregionsand improves the robustnesswith respectto the presenceof noise.As matterof fact, it giveslargeruniform regions,whosecharacterizationin termsof TPM is morereliablecomparedto regionswhosesizeapproachesthepixel’sone.Furthermore, thedatalikelihooddescriptionis basedonamultivariateGaussianmodelingwhich takesintoaccountthecorrelationinthecolor space. Finally, its treestructuredformulation,similar to that of the tree-structuredvectorquantizationalgorithm[11], speedsup theprocessing,andallows for adaptively choosingthecolor-clusterto be split in a recursive process.Sucha featuremay be usedto make uniform the sizeoftheregionelementsprovidedby thecolorclustering,asto ensureauniformspatialresolutionof themethod.

Thenumberof regionsto besingledout needsto beeitherfixeda priori, on thebasisof someheuristicrule, or estimatedaccording to someclustervalidation criterion. To this end,a trade-offshouldbe taken into account. In fact, a high numberof regionsproducedat this stage,althoughwould helpto preserve imagedetails,will producetoo smallelementaryregionswhosesubsequentspatialcharacterizationfor next processingwouldbelessreliable.Also,eventuallongerrangespatialinteractionsof a giventexturewould needmorestepsto propagatein thefinite-statemodeling,andhenceaskfor amorecomplex optimizationprocessto berecognized.

Beingawareof theseaspects,we decidedto usea simpleheuristicrule, to be refinedin futurework, asat presentotheraspectsof the algorithmseemto be morecritical andnecessitateto beaddressedfirst. Onthebasisof ourexperimentalobservations,wefoundthedoubleof themaximumnumberof texturesexpectedin theimageto bea reasonablechoicefor thenumberof color clustersto beextracted.This canbe intuitively justifiedby the fact thatany non-trivial texturehasat least

RR n° 6 0 6 6


two modesin thecolor space.Hence,we areensuringthat, in average,we have at leasttwo colorspertexture.

3.4 Spatial-basedclustering (SBC)

In theprevioussectionwedescribedhow acertainnumberof partially-definedstates,or color-states,arederived from the imageby CBC discriminatingonly w.r.t. color. The spatial-basedclustering(SBC)thenfocuseson thefurthersplit of thesestatesby usingthecontextual informationcontainedin theTPMsdefinedin Section 3.1.

In principle, this splitting processshouldbe carriedout simultaneouslyfor all the color-states,asthecharacterizationof thefully-defined stateswouldnot referto partially-definedstates.In otherwords, while computingthe TPM of a region of a given color, one shouldrefer to fully-definedstates,requiringtheother color-statesto bealreadysplit. Therefore, it is clearthattheoptimizationproblemmaybeaddressediteratively by alternatingstateestimation andregion labeling. Howevertwo mainconcernsaboutthissolutionhaveto benoticed.Thefirst oneis thatwehavenoguaranteesaboutthe convergenceof suchan iterative process.The secondoneis aboutthe reliability of thecharacterizationof thestates.While the former is easilyunderstoodandperhapscanbeaddressedin practice,thelatteris morecritical andneedsto befurtherclarified.

For this purposeconsidertheexamplebelow. Let 24 bethenumberof color-statescomputedbyCBC,andsupposetheCBC outputmapto have around104 connectedregions,i.e. imageelementsto be characterizedandclustered.Also let 12 be theaveragenumberof offspringstatespercolor-state.Hencethetotalnumber of statesis 24×12 = 288, andtheTPMswil l contain288×8 = 2304elementswith an high degreeof sparseness.This meansthat we shouldperforma clusteringin aspacewhere theratiobetweenthenumber of elementsandthedimensionalityof thespaceis around104/2304 ≈ 4, which is clearly not enoughand,then,a considerablefeaturereductionis needed.Furthermore,aneventualfeaturereductionshouldbeiteratedaswell astheclustering,causinghugecomputationsandmakingeven morecritical theconvergenceof theprocess.

For thesereasonsweimplementedadifferentoptimizationschemewhichdoesnotrequireajointsplit of color-states.In practiceweconsideranapproximatecharacterizationof theimageelements,by simplyreferringto thepartially-definedstates.6 In thiswaywefactorizetheoptimizationprocess,sincethesplitsof thecolor-statesaremutuallyindependent.Nevertheless,thedimensionalityof thefeaturespaceis still largefor a reliablecharacterization,andthenwe useto a principal componentanalysis(PCA) to reduceit.

In particulara singlesplit, which is a clusteringof connectedregionswith thesamecolor-state,sayω, is carriedout asfollows. For eachconnectedelementn ∈ 1, . . . , Nω, thecorrespondingTPM Pn

ω is computedby Eq.5. Experimentallywe found moreeffective the useof a logarithmictransformationQn

ω definedas:

Qnω(ω′, j)

=

log[1 − Qnω(ω′, j)], ω′ = ω

log[Qnω(ω′, j)/(1 − Qn

ω(ω, j))], ω′ 6= ω.(7)

6W.r.t. thepreviousexample,only 24 × 8 TPMsarenow involved.

INRIA


Apart theuseof thelogarithm,notice that for ω′ = ω this correspondsto replacetheprobabilityofkeepingthecurrentstatewith its complement(i.e. probabilityto leavethestate).When ω′ 6= ω, thatis thecasewhenstateω is left for ω′, we just conditionedw.r.t. theevent“ω is left”.

Now for eachfixedω′ considerthe8-dimensionalrow vectorsQnω(ω′, ·), 1 ≤ n ≤ Nω, andre-

ducethemto scalarvaluesby PCA,i.e. by takingtheprojectingon thelargesteigenvector. Roughlyspeakingwearejustconsideringanaveragebehavior w.r.t. thespatialdirections,wherethosewhicharemore“informative” are weightedmore. By doingsoindependentlyfor eachrow ω′, we reducethematricesQn

ω to a |Ω|-dimensionalcolumnvector, whichis thenfurtherreducedby PCA,asto en-surea smallerdimensionality. In particular thenumberof meaningfulcomponentsis automaticallychosento keep75%of theenergy. ThissecondPCA is justifiedby thefactthatsomeneighborstatesactuallyoccurrarelyandaredueto thenoise.Thereforea PCA improvesfurther therobustnessoftheclustering.

Finally, the clusteringis performedby applyinga k-meansalgorithmin this featuresubspacewheredataareno longerassparseas in the full spacespannedby the TPMs. The numberK ofsubclusterspereachcolor-statesplit is fixedapriori, sinceits exactcomputationis notnecessary. Inparticular, K mustbelargeenoughtoavoidunder-clustering, thateventuallycouldcausethemergingof differenttexturesduring thesubsequentregion merging step.On theotherhand,a too largeK,henceanover productionof states,maycausea singletexture to besplit afterwards.Accordingtoourexperience,agoodcompromiseis to fix K to theexpectednumberof texturein theimage,or toits upperboundif this is theonly known information.

3.5 Regionmerging: the regiongain

Theresultof thesequenceof steps describedabove (CBC andSBC), is a partitionof the imageinregions,eachof themcorrespondingto a statedefinedin termsof color, shapeandspatialcontext.Accordingto the hierarchicalmodeling formulatedabove (see Section3.1), thesearethe terminalstates(finestscale)whichhavenow to berelateduntil all collapsein themacrostateassociatedwiththehierarchy root,i.e. with thewhole image(coarsestscale),whichcorrespondsto arecursiveregionmerging. Theaim of this processis to collect togetherstates,i.e. regions,thatbelongto thesametextureasto obtainthe texture identificationandsegmentation.Now, given thata singlestatemayinteractwith boththestatesof thesametextureandthoseof aneighboringone,whatis importantisto privilege themerging of statesstronglycorrelated,asthey likely belongto thesametexture. Inthis way the intra-texture mergingswill be privilegedw.r.t. the inter-texture onesthat, eventually,will appear at thetop levelsof thehierarchy. In particular, a correctstructureidentificationrequiresall intra-texture mergings to be performedbeforeany inter-texture one. If we achieve this, thenall marginal texturemodelsareenclosedasnon-overlappedsubstructuresin theoverall hierarchicalmodel,hencea separationof them is possibleby simply stoppingthemerging processwhenall theintra-texturemergingsaredone.

As alreadydiscussedabove, we focusin this reporton binaryhierarchiesin orderto reducethecomplexity, aswell as to benefitof the greater flexibility of a binary solution for the purposeofrepresentation.Thereforethestructurebuilding canbeperformedby meansof a recursivesequenceof binarymergingsdrivenby a testparameter. Dueto theabove remark,wehaveproperlydefineda

RR n° 6 0 6 6


test,namelytheregion gain, which is definedin thefollowing. At any givenstept of therecursivemerging sequence,let Ω(t) be the currentsetof stateswhich arecandidatesfor the next merging.ThenΩ(0) is thesetof terminalstates,while Ω(L−1) is just therootstate,with L = |Ω(0)|. Now, foreachstatei ∈ Ω(t), correspondingto regionRi, theregiongain is definedas:

Gi =p(s ∈ Ri)

maxj 6=i p(r ∈ Rj |s ∈ Ri)(8)

= p(s ∈ Ri) ·1

p(r /∈ Ri|s ∈ Ri)·

p(r /∈ Ri|s ∈ Ri)

maxj 6=i p(r ∈ Rj |s ∈ Ri)(9)

wheres is animagesiteandr is any of theeightneighborsof s.Noticethatthetestparameterisnotameasureof similaritybetweenstatesbutameasure(inverse)

of the migrationspeedof statei toward otherstates,in particularthe mostattractive state,j∗ =arg maxj 6=i p(r ∈ Rj |s ∈ Ri). In fact,it is a ratio betweentheareacorrespondingto statei, whichcanbeconsideredasaweight, andtheprobabilityto leavestatei for j∗ (Eq.8).

Eventually, at current time t the weakest statei, that is associatedto the minimum gain, isabsorbedby thestatej∗ thatattractsit themost.Thenthemerging processis iterateduntil therootof thehierarchy is reached.

Thefactorizationin Eq.9,allowsusto giveaneasyinterpretationof thegain,asthespatialscaledependenceof the outputhierarchy. Indeed,the gain of a stateis alsoan indicatorof the scaleofits correspondingregion,andconsistentlyit typically increaseswhenmerging statesandassociatedareas.Thefirst factoris clearly relatedto thescalesinceit is proportionalto thearea,aswell asthesecond,which indicatesthecompactnessof theregion. In fact,thelargertheprobability to leave thestate(denominator),the larger theregion perimeter, thenthelesscompacttheregion is distributed.The third factor, which is just the inverseof theoccurrenceof theneareststategiven that thestateis left, is insteadrelatedto the spatialcontext rather thanto the scale. For a fixed region, in fact,the presenceof a dominant neighborratherthanseveral equallyoccurringones,indicatesa clearrelationshipbetweentwo stateswhich likely have to bemerged.

Recallthatoncethecompletesequenceof merging is definedthenanestedhierarchical segmen-tation is given. Therefore,any usermay selectthepropersegmentationhe/sheneedsdependingonthepurpose.Sometimes,instead,theuserhasadditionalinformationthatmaydrive thevalidationin a post-processing.Nevertheless,this problemcanbe addressedjust in terms of scale,sincetheregion gain is directly relatedto it andit is sufficient to give again threshold.In this sense, fixingthescalecorrespondsto take areferenceregion with fixedarea(sayα × |S|), shape(saya square)andcontext (sayonly onesurroundingneighbor).With theseconstraintsthegivenregion will haveapproximatively againequalto:

G∗ =8α(

√

α|S| − 2)2

12√

α|S|, (10)

whichcanbeusedasa threshold throughthespecificationof α.Concerningthecomputationalloadof themerging process,it is easyto recognizethat it is very

light, sinceoneonly hasto computethe TPMs for the terminal statesprovided by SBC,andthen

INRIA


keepan orderedlist of the gains of the currentstateswhile storing the structuredefinedby themergesequence.Also, theTPMsof merging statesarejust combinationsof otherTPMs(hencenocomputationatpixel level), andthegainsarederivedfrom theTPMsaswell.

3.6 Enhancedregiongain

Theregion gain definedabove measureshow likely theregion is itself thesupportof a texturew.r.t.the hypothesisthat it is just a part of a larger support. Whenthe gain is low we let the region beabsorbedfrom the“nearest”,asto obtaina largerone.By definitionof theregiongain, the“nearest”meanstheneighboringregion thatsharesthelargestboundarywith thegivenregion. Althoughthiscriterion has provided goodresults,therearecertaincaseswhereit fails. Indeed,the presence ofnoisemayincreasethelengthof theboundarybetweentwo regionsandmakethemcloseraccordingto thegaindefinition.Thisproblemoccursoftenbecauseof theboundaryfragmentationphenomenacausedby colorquantizationduringtheCBC step.

In order to reinforce the measure,as to improve the robustness,we considerednot only thedegreeof contactbetweenregionsbut also their spatialdistribution similarity. To do so we haveintroducedanadditionalterm in thegain, that is theKullback-Leiblerdivergence(KLD) [18]. TheKLD betweentwo distribution,p andq, is definedas:

D(p‖q)=

⟨

logp(x)

q(x)

⟩

p

=

∫

p(x) logp(x)

q(x)dx, (11)

where〈·〉p is thestatisticalaverageaccordingto thedistribution p. SinceD(p‖q) is theaveragelog-likelihoodratiobetweenp andq, thenit isameasureof theinefficiency of assumingq in placeof p.Henceit is well adaptedto describe how muchtwo objectsareclosew.r.t. their spatiallocations. Inparticular, namedqi(x) thedistributionof thespatial locationof region i, wherex is the2-D spatialposition,thenthemodifiedregiongainGi

KL of statei is definedby:

log GiKL

= min

j 6=i

logp(s ∈ Ri)

p(r ∈ Rj |s ∈ Ri)+ D(qi‖qj)

(12)

= minj 6=i

logp(s ∈ Ri)

p(r ∈ Rj |s ∈ Ri)+

⟨

logqi(x)

qj(x)

⟩

qi

, (13)

wherewe referredto the logarithmic formulation to properlycombinethe previous gain with theKLD term.Noticethatby removing theKLD termthegain reducesto theoriginalone.

Thecomputationof theKLD is in generalquitedifficult for mostof thedistributions,andin afew casesadmits a closedform. Onesuchcaseis that of two Gaussiandistributionsp andq forwhich thedivergenceis givenby [22]:

D(p‖q) =1

2

(

log|Σq|

|Σp|+ tr(Σ−1

q Σp) + (µp − µq)T Σ−1

q (µp − µq) − d

)

(14)

wherep ∼ N (µp,Σp), q ∼ N (µq,Σq) andd = 2 is the distribution dimensionality. Due to itssimplicity, theabovemodelinghasbeenconsideredfor comparingthespatialdistribution of differentregions,andusedfor thenew regiongain (Eq.12).

RR n° 6 0 6 6


4 Experimental Results

4.1 The PragueTextureSegmentationDatageneratorBenchmark

The Praguetexture segmentationbenchmark[14], developedby the Pattern Recognition researchgroupof the Instituteof InformationTheoryandAutomation,CzechAcademyof Sciences,hasatwo-fold objective:

• to mutuallycompareandrank differenttexturesegmenters(supervisedor unsupervised),

• to supportnew segmentationandclassificationmethoddevelopments.

In particulartheserverallows:

• to obtaincustomizedexperimentaltexturemosaicsandtheir correspondinggroundtruth,

• to obtainthebenchmarktexturemosaicsetswith their correspondinggroundtruth,

• to evaluateany working segmentationresult and compareit to the resultsobtainedby thestate-of-the-artalgorithms,

• to include any algorithm (reference,abstractand benchmark results) into the benchmarkdatabase,

• to checksinglemosaicevaluationdetails(criteriavaluesandresultedthematicmaps),

• to ranksegmentationalgorithmsaccordingto themostcommonbenchmarkcriteria,

• to obtainLaTeX codedresulting criteriatables.

The computergeneratedtexture mosaicsandbenchmarksprovided by the server arecomposedofthefollowing texturetypes:

• monospectraltextures,

• multispectraltextures,

• BTF (bidirectionaltexturefunction)textures,

• rotationinvarianttextureset (work in progress),

• scaleinvarianttextureset(work in progress).

Furthermore,all generatedtexture mosaicscanbe corruptedwith additive noise,andtraining setsaresuppliedin asupervisedmode.

INRIA


4.1.1 Performanceassessment

Thebenchmarkserverprovidesa comparativeanalysisof all theresultsuploadedby usersaccordingtoseveralaccuracy indicatorswhicharegroupedin threemaincategoriesasdetailedin thefollowing.

Region-basedcriteria

Theregion-basedcriteria[16] mutuallycomparethemachinesegmentedregionsRi, i = 1, . . . ,Mwith thecorrectgroundtruthregionsRj , j = 1, . . . , N . Theregionsoverlapacceptanceis controlledby thethresholdk = 0.75 (0.5 < k < 1). Singleregion-basedcriteria aredefinedasfollows:

• CS (correctdetection):[Rm; Rn] iff

(i) cardRm ∩ Rn ≥ k cardRm

(ii) cardRm ∩ Rn ≥ k cardRn

• OS (over-segmentation):[Rm1, . . . ,Rmx; Rn], 2 ≤ x ≤ M iff

(i) ∀i ∈ 1, . . . , x, cardRmi ∩ Rn ≥ k cardRmi

(ii)∑x

i=1 cardRmi ∩ Rn ≥ k cardRn

• US (under-segmentation):[Rm; Rn1, . . . , Rnx], 2 ≤ x ≤ N iff

(i)∑x

i=1 cardRm ∩ Rni ≥ k cardRm

(ii) ∀i ∈ 1, . . . , x, cardRm ∩ Rni ≥ k cardRni

• ME (missed):[Rn] iff

(i) Rn /∈ correctdetection

(ii) Rn /∈ over-segmentation

(iii) Rn /∈ under-segmentation

• NE (noise):[Rm] iff

(i) Rm /∈ correctdetection

(ii) Rm /∈ over-segmentation

(iii) Rm /∈ under-segmentation

Pixel-wiseweightedaveragecriteria

Let usdenote

ni,• =K

∑

j=1

ni,j and n•,i =K

∑

j=1

nj,i (15)

RR n° 6 0 6 6


whereK is a number of classes(or regions),n is the numberof pixels in the testset,ni,j is thenumberof pixels interpretedasthe i-th classbut belongingto the j-th class. ı is either i for su-pervisedtestsor mappingof the i-th classground truth into aninterpretation segmentbasedon theMunkresassignmentalgorithm[21] for unsupervisedtests. The following pixel-wisecriteria areimplemented:

• O (omissionerror, theoverall ratio of wrongly interpretedpixels):

O =1

n

K∑

i=1

Oi =1

n

K∑

i=1

(n•,i − nı,i) ∈ [0, 1], (16)

whereOi is thei-th classomissionerror.

• C (commissionerror, theoverall ratioof wronglyassignedpixels):

C =1

n

K∑

i=1

Ci =1

n

K∑

i=1

(nı,• − nı,i) ∈ [0, 1], (17)

whereCi is thei-th classcommissionerror.

• CA (theweightedaverageclassaccuracy):

CA =1

n

K∑

i=1

nı,in•,i

n•,i + nı,• − nı,i

∈ [0, 1] (18)

• CO (recall- theweightedaveragecorrectassignment):

CO =1

n

K∑

i=1

n•,iCOi =1

n

K∑

i=1

nı,i ∈ [0, 1], (19)

• CC (precision,objectaccuracy, overall accuracy):

CC =1

n

K∑

i=1

n•,iCCi =1

n

K∑

i=1

nı,in•,i

nı,•∈ [0, 1], (20)

• I (typeI error):

I =1

n

K∑

i=1

(n•,i − nı,i) = 1 − CO ∈ [0, 1], (21)

• II (typeII error):

II =1

n

K∑

i=1

nı,•n•,i − nı,in•,i

n − n•,i

∈ [0, 1], (22)

INRIA


• EA (meanclassaccuracy estimate):

EA =1

n

K∑

i=1

2nı,in•,i

n•,i + nı,•∈ [0, 1], (23)

• MS (mappingscore,emphasizes theerrorof not recognizingthetestdata):

MS =1

n

K∑

i=1

(1.5nı,i − 0.5nı,•) ∈ [−0.5, 1], (24)

• RM (rootmeansquareproportionestimationerror):

RM =

√

√

√

√

1

K

K∑

j=1

(

nj,• − n•,j

n

)2

=1

n

√

√

√

√

1

K

K∑

j=1

(Cj − Oj)2 ∈ [0,∞), (25)

indicatesunbalancebetweentheomissionOi andcommissionCi errors,respectively.

• CI (comparisonindex, includesboth typesof errors):

CI =1

n

K∑

i=1

nı,i

√

n•,i

nı,•=

1

n

K∑

i=1

n•,i

√

CCiCOi ∈ [0, 1], (26)

whereCCi, COi arethe objectprecisionandrecall. CI reachesits maximumeitherfor theidealsegmentationor for equalcommissionandcommissionerrorsfor every region (class).

Consistency errorcriteria

LetS1, S2 betwo segmentations,R1,i is thesetof pixelscorrespondingto aregionin theS1 segmen-tationandcontainingthepixel i, |R| is thesetcardinalityand\ is theset difference.A refinementtolerantmeasureerrorwasdefined[19] ateachpixel i:

ǫi(S1, S2) =|R1,i \ R2,i|

|R1,i|. (27)

Thisnon-symmetriclocalerrormeasureencodesameasureof refinementin only onedirection. Twoerrormeasuresfor theentire imagearedefined:theGlobalConsistency Error, GCE, forcesall localrefinementsto bein thesamedirectionwhile theLocal Consistency Error, LCE, allows refinementin bothdirections.

• GCE (globalconsistency error):

GCE(S1, S2) =1

nmin

∑

i

ǫi(S1, S2) ,∑

i

ǫi(S2, S1)

(28)

RR n° 6 0 6 6


• LCE (local consistency error):

LCE(S1, S2) =1

n

∑

i

minǫi(S1, S2) ,∑

i

ǫi(S2, S1) (29)

LCE,GCE ∈ [0, 1], LCE ≤ GCE (30)

4.1.2 Comparativesegmentation algorithms

Thanksto thePraguebenchmarksystemwe benefitof severalsegmentationresultswhich allow usto make acomparativeanalysisof ourmethod.Thedifferentalgorithmswhichhavebeenrunon thesamebenchmarkdatasetsarelistedandbriefly describedbelow:

• GMRF (GaussMRF model)with EM [12]:Singledecorrelatedmonospectraltexture factorsareassumedto be representedby a setoflocalGaussianMarkov randomfield (GMRF) modelsevaluatedfor eachpixel centeredimagewindow and for eachspectralband. The segmentationalgorithm,basedon the underlyingGaussianmixture(GM) model,operatesin thedecorrelatedGMRF spaceof parameters.Thealgorithmstartswith an oversegmentedinitial estimationwhich is adaptively modifieduntiltheoptimalnumberof homogeneoustexturesegmentsis reached.

• AR3D (3-D Auto Regressivemodel)with EM [13]:This algorithmis similar to thepreviousone,but theGMRF modelingis replacedby a threedimensionalauto-regressivemodel.

• JSEG[8]:Themethodconsistsof two independentsteps,colorquantizationandspatialsegmentation.Inthefirst step,colorsin theimagearequantizedtoseveralrepresentativeclassesthatcanbeusedtodifferentiateregionsin theimage.Theimagepixelsarethenreplacedby theircorrespondingcolorclasslabels,thusformingaclass-mapof theimage.Thesubsequent spatialsegmentationstepappliesto theclass-map,asto obtain an image,namelythe “J-image”,wherehigh andlow valuescorrespondto possibleboundariesandinteriorsof color-textureregions.A regiongrowing methodis thenusedto provide the final segmentationon the basisof a multiscaleJ-images.

• Blobworld (asystemfor segmentationandCBIR) [1, 4]:Thealgorithmwhich wewill referto asBW is thebasicsegmentationtool usedin thecontent-basedimageretrieval systemblobworld [4]. Eachimageis segmentedinto regionsby fittinga mixtureof Gaussiansto thedatain a joint color-texture-positionfeaturespaceby meansofanEM algorithm.Eachregion (“blob”) is thenassociatedwith color andtexturedescriptors,wherethetexturalfeaturestakenin considerationarecontrast,anisotropy andpolarity. Finally,theoptimalnumberof Gaussiancomponentsis automaticallyselectedby meansof theMin-imum DescriptionLength (MDL) criterion. Furtherdetailsaboutthesegmentationalgorithmcanbefoundin [1].

INRIA


• EDISON(EdgeDetectionandImageSegmentatiONsystem)[5]:This algorithmis basedon the fusion of two basicvision operations,that is imagesegmen-tation andedgedetection, the former basedon global evidence,the latter focusedon localinformation. This integrationis realizedby embeddingthe discontinuity(edge)informationinto to theregionformationprocess,andthenusingagainit to controlapost-processingregionfusion. In particularEDISONcombinesthemean shift basedsegmentation[6] with a gener-alizationof thetraditionalCanny edgedetectionprocedure[3] which employs theconfidencein thepresenceof anedge[20]. For moredetailsabouttheEDISONalgorithmsee[5].

4.1.3 Texturemosaicsegmentation results

Two versionsof the proposedsegmentationmethodweretestedon thedataset,referredto asTFRandTFR+, which areassociatedwith the two definitionsof region gain, seeEq.8 andEq.12re-spectively. The choiceof the settingparametersfor the two implementationswasthe same. Thenumberof partially-defined(color) stateswas24, that is the CBC/TS-MRFperformeda 24-classcolor-basedsegmentationfor all theimages.Thenumberwasnotoptimizedbut just chosenaccord-ing to the heuristicrule that takesthe doubleof the maximumnumberof expected texturesin theimage,assuggestedin Section3.3. Actually for all theimagesthenumberof texturesis alwayslessthan12, so we assumedthis informationto be known andfixed the parameterto 24. Indeed,wehave run sometestswith differentnumbersof quantizationcolorsandfoundonly slightly differentresults. The samemaximumnumberof expectedtexturesis alsousedasK in the subsequentk-meansclusteringsduringtheSBCstep,andsimilar considerationsaboutreliability hold in this caseaswell.

The benchmark dataset is composedof twenty different 512 × 512 texture mosaics, ten ofwhich areshown in Figures5-14 togetherwith the associatedground-truthandthe segmentationsperformedby thedifferentmethodsabove mentioned.Thenumerical comparison,accordingto thebenchmarkcriteria(seeSection4.1.1),is summarizedin Tab.1.

Thesystemprovidesacomparisonw.r.t. a largenumberof indicators,someof whichareregion-based,someothersarepixel-wiseaccuracy indicators,anda few of themgive ameasureof consis-tency. A completedescriptionof all theparameters,aswell asall theresultspresentedhere,canbefoundon thesystemwebpage[14].

Theinterpretationof thenumericalresultsin Tab.1mayappearambiguousin someregards,since(of course)no algorithmoutperformsuniformly all theothers.However it canbeeasilyrecognizedthat the two versionsof TFR seem to outperformthe otheronesin mostof the cases,with TFR+beinggenerallybetterthanTFR. Indeed,thevisualinspectionof thedifferentsegmentationsshownin Figures5-14 allows an easierinterpretation,showing clearly the superiorperformancesof theTFRalgorithms,especiallywhenthetexturespresentvery low frequency patterns,whichareknownto bemoredifficult to reveal.

The mostevident drawbackof the referencemethodsis the tendency to over-segment,ascanbededucedby visually inspectingthesegmentations,andconfirmedalsoby theover-segmentationindicatorOS (seeTab.1). At theopposite,TFRhasaslight tendency to under-segment,US = 23.99(seeTab.1), w.r.t. theotherones,while TFR+ hasthemostbalanced behaviour, keepingquite low

RR n° 6 0 6 6


Texturemosaic Ground-truth JSEG

Blobworld EDISON AR3D

GMRF TFR TFR+

Figure5: TexturemosaicNo.1: data, ground-truth and several segmentations.

INRIA