Dynamic Scale-Space Theories

Alfons H. Salden*, Bart M. ter Haar Romeny, and Max A. Viergever

Image Sciences Institute, Utrecht University Hospital, Heidelberglaan 100, 35S4 CX Utrecht, the Netherlands

Abs t r ac t . Image formation of a two-dimensional input image can be quantified by imposing an image induced connection and computing the associated torsion and curvature. The latter aspects of image formation are especially nonvanishing at sets of discontinuities and non-isolated singularities, such as ridges and ruts. Next dynamic scale-space theories for the input image are constructed on the basis of an image induced connection. Finally dynamic scale-space theories for the image formation are constructed that are coupled to the image formation itself.

1 I n t r o d u c t i o n

The goal of existing scale-space theories [1] is to find a stable and reproducible description of an input image, that is slightly corrupted by noise above some level of scale and is invariant under certain t ransformation groups. Normally one assumes tha t the connection on the image domain is flat, i.e. the image domain is conceived as a Euclideanl affine or projective space. However, a connection by the input image [2] is characterised by a non-zero torsion and curvature nicely revealing the image formation, i.e. the total of physical ac- tions applied to a ground state yielding the input image. Such a torsion and curvature one identifies in defect theory [3] and gauge field theories [4] with a Burgers vector field and Frank vector fields, respectively. These vector fields are in particular non-zero along so-called defect or cut lines where physics becomes multi-valued. The question arises whether one can detect such lines also in image formation and whether one can smooth the input image or the image formation as much as possible within them.

Our aim is to demonstrate in section 2 that a grey-valued input image defined on a two-dimensional Euclidean space can be provided with an im- age induced connection tha t allows torsion and curvature of image formation to be read out. I t is shown tha t the sets of discontinuities and non-isolated singularities, and in particular ridges and ruts of such an input image are topological equivalent objects corresponding to particular defect or cut lines in image formation. In section 3 image induced connections are used to for- mulate dynamic scale-space theories for the input image. Furthermore, the image induced connection and its associated torsion and the curvature are

* This work was supported by the Netherlands Organisation of Scientific Research, grant hr. 910-408-09-1, and by the European Communities, H.C.M. grant hr. ERBCHBGCT940511.

249

used to formulate dynamic scale-space theories for the image formation itself. Dynamic scale-space theories compared to other scale-space theories do take the aspects of image formation latent in the torsion, curvature or topological characteristics such as local connectivity into account to steer these aspects themselves. Whether or not the input image is static or spatio-temporal is irrelevant for the idea behind dynamic scale-space theories. This does not mean that the particular physical realisation of a spatio-temporal domain as an array of detectors distributed over space-time will not be committed to certain symmetries that are imposed by nature in terms of e.g. a connection on that network. After the network has been subjected to image formation the latter formation is scaled and not the formation normally induced by nature.

2 M o d e r n G e o m e t r y o f I m a g e F o r m a t i o n

As in the sequel the generalisation of the existing scale-space theories [1] heavily relies on modern geometry its most important ingredients are sum- marised. In subsection 2.1 differential geometry and in subsection 2.2 integral geometry are briefly treated, and applied to certain computer vision prob- lems. For more extensive treatments the reader is referred to [5]. Note that the geometries considered are not only confined to Riemannian ones.

2.1 Different ial G e o m e t r y

Let M be a D-dimensional image domain parametrised by canonical coordi- nates p = (pl, . . .pD). Now consider the frame bundle F = P(M, 7r, A(D, IR)) where P is the total space consisting of all frames ~p at each point p E M, 7r: P --+ M is the projection and A(D, IR) = GL(D, IR) t> T(D, IR) the full affine group, where GI(D, IR) is the general linear group and T(D, JR) the translational group. In this context let's define a local frame as follows.

Def in i t ion 1. A local frame ~p is defined by:

G ~- ( e 0 ; e l ' ' ' ' ' e D ) (P) '

where the vectors (e0, e l , . . , eD)(p) span the local tangent space TpA(D, IR).

Now an affine connection F in the frame bundle F is defined as follows.

Def ini t ion 2. An affine connection F in the frame bundle F is defined in terms of the Lie algebra 6(D, ]R)-valued connection one-forms (w~, w~) and the frame vectors (e0, e l , . . , eD) through the following equality:

V r 0 ~ odiei, ~ e i J ~- bdicj ,

where V is the covariant differential operator.

250

The affine connect ion/" satisfies so-called structure equations:

T h e o r e m 3. Given an affine connection F in the frame bundle F, defined in (2), then the connection one-forms satisfy the following structure equations:

D~ i = dw i + w~ A wk = ~i Dw] = dw} + w~ A w~ = ,].,

where d the ordinary exterior derivative, A is the wedge product, D the co- variant derivative, f?i is the torsion 2-form and f~ is the curvature 2-form.

Proof. See [5].

In turn the torsion and the curvature 2-form satisfy so-called Bianchi identi- ties:

T h e o r e m 4. Let F be an affine connection in the frame bundle F with tor- sion 2-form f?~ and curvature 2-form f~. The integrability conditions for the structure equations, that are the Bianchi identities, are given by:

Dr? i = f2~ A w j, Df?~ = O,

Proof. See [5].

Normally the above connection F is not compatible with a metric "y on manifold M [5].

D e f i n i t i o n 5. Let F be an affine connection in a frame bundle F , and 7 a metric structure on the manifold M. The metric 7 is compatible with the connection % if and only if,

V7 = 0,

where the covariant derivative V is consistent with the connection P.

After this rather brief summary of the essential ingredients of differential geometry let us describe the image formation of a smooth grey-valued input image L0 in terms of modern differential geometry.

D e f i n i t i o n 6. A smooth grey-valued input image Lo on two-dimensional Euclidean space E 2 onto the space IR + of real-valued grey-values is defined by a scalar-valued density function:

L0 : E 2 -~ lR + �9

Furthermore, let us confine ourselves first to those objects in the image domain E 2 represented by the input image that are invariant under the spa- tially homogeneous Euclidean group E(2) and under the group of arbi t rary non-constant spatially homogeneous grey-value transformations f : L0 -4 f (Lo) . Imposing invariance under the two groups of t ransformations above the sought objects coincide with those that can be constructed on the basis of the well-known isophotes and flowlines of the input image.

251

D e f i n i t i o n 7. An isophote Ci of a two-dimensional grey-valued input image L0 is defined by:

c~ : {x �9 E21L0(~) : ~ �9 ~{0+ }.

D e f i n i t i o n 8. A flowline CI of a two-dimensional grey-valued input image L0 is defined by:

E21 dx(p) OLo } CS = x �9 @ - Ox--n (x(p)) ,

where p E IR is an arbitrary parameter.

Now various objects with a non-trivial image induced connection can be constructed.

Example 9. Construct the following Euclidean differential geometry on the net of isophotes and flowlines by choosing a frame field @ and a connection ( ~ , ~ ) :

r = (e , , e2),

k--~'k--sk ] j - - ( - - 1 ) k + 1 ~ k 0 '

where el, s 1, al and e2, 8 2, n2 are unit tangent vector fields, the Euclidean arclengths and curvatures on the isophotes and flowlines, respectively (realise tha t the latter curvature are just invariant zero-forms being factors in the choice of connection). It is readily shown by means of the Cartan structure equations that the net of isophotes and flowlines with the above connection has both a non-zero torsion tensor T and a non-zero curvature tensor T~:

T = Tjk ~w j | ~k | ~i,

~r~ --__ Rikl jwi | 03k (~ Ojl | ej,

where

1- F i Tjk ~ = ~( J~ - G j ~ ) ,

�9 d d / . �9 R j ~ ~ = d-GFj~ i - ~ ~ i + Fj~ ~ F m ; - G~ nFnj ~-

Note that for each isophote or flowline associated a fiat connection one will observe that both the torsion and curvature tensor are identically zero.

Another question frequently posed is which objects of the net in Exam- ple 9 are invariant under the group of diffeomorphisms of the image domain E ~ and under the group of arbitrary non-constant spatially homogeneous

252

grey-value transformations f : Lo --+ f (Lo) , i.e. the group of anamorphoses i. This group is evidently not preserving the total grey-value of the input image nor the Euclidean differential geometry of the net of isophotes and flowlines quantified in Example 9. Despite the latter inconvenience requiring invari- ance under the group of anamorphoses has the advantage to point out those objects in the image across which the image formation abruptly changes. Moreover, changes of view or lightning, and dynamics of an object can better be modelled on the basis of these objects than on basis of the input image.

It 's clear that the set of (non)-isolated singularities and the set of disconti- nuities of L0 remain the same topological equivalent sets under anamorphoses. The vanishing of the image gradient is not affected by any anamorphosis, nei- ther are discontinuities in L0. A set of nonisolated singularities occurs, for

example, for input images L0(x, y) = ~ ( ( x +zy)" ~ 2 ) , x n E I N .

Not so obvious is that the landscape of ridges and ruts of the input image L0 [6], i.e. the set of the singular flowlines, are also not affected by anamor- phoses. The latter so-called topological equivalence of the landscape of ridges and ruts can be described by the fact that across ridges and ruts the flowlines have opposite convexity. Consequently the connection at the ruts and ridges is completely degenerate meaning that any order of derivative with respect to the Euclidean arclength parameter s 1 of the flowline curvature is vanishing. Because of the fact that to a finite order there will always be non-ridge or non-rut points for which they are zero and one has to take all orders into account, it is impossible to distinguish on the basis of a pure local analysis between ridges, ruts and the borders of their influencing zones consisting of e.g. inflection points. Nevertheless, possible ridges and ruts can be discerned on the basis of the isophote curvature hi- If ni > 0 and n2 = 0, then the points belong to the set of possible ridge points. If/~1 <~ 0 and n2 = 0, then the points belong to the set of possible rut points.

The reader might object that these objects are nothing more than some special sets of discontinuities of the multi-local second order jet structure and that this multi-local property not only occurs for flowlines in particular. In- deed, the change in convexity is an image induced discontinuity determined by multi-local properties of the exterior derivative field of the image influ- enced by second order jet information. And for isophotes such changes in convexity do also occur. For example, across the set of non-isolated singular- ities along the flowlines of the input images mentioned above the isophotes also change convexity. However, this set can be considered, equally well as the set of extrema of the input image, as just parts of the landscape of ridges and ruts.

1 A chaJage of view or lightning can in general not be captured by the group of anamorphoses, because each input image is an observation of an intensive phys- ical entity. They yield, even though they might preserve the total grey-value fallen onto the detector array, topological transformations of the objects that are invariant under the group of anamorphoses [2].

253

At a n-junction where different components of a set of nonisolated singu- larities or ridges and ruts are coming together there is just a n-fold branching or a n-fold degeneracy of the image gradient, respectively.

In defect and gauge field theory [8,4] these sets of different types of sin- gularities are identified with so-called defect or cut lines along which the physical properties of a medium change abruptly. In the context of computer vision or pattern recognition one can conceive these lines as the borderlines along which on each side other types of image detail (texture) are inserted or removed, or image formation processes have taken place. One can state that image formation concerns a non-differentiable or better a topological trans- formation of the landscape of isophotes and flowlines of a ramp function.

In order to actually find still on a quasi local basis ridges and ruts parametrise the input image by means of e.g. the x2-axis and trace the extrema of the image gradient length on each line for which x 2 is constant. Local minima and maxima in the image gradient length on these lines then correspond to rut and ridge points, respectively. Alternatively, take a strip of thinkness of one just noticeable isophote and walk around the global maximum and keep track of the extrema of the length of the image gradient field upon encircling it with the next just noticeable isophote. This supplies us with a so-called topological geometric method for finding ridges and ruts [2].

2.2 I n t e g r a l G e o m e t r y

Following Cartan [5] one can apply a Euclidean displacement to determine the translation vector field and the rotation vector fields to operationalise the torsion and the curvature of the frame bundle F with connection F.

D e f i n i t i o n 10. Let F be a connection in the frame bundle F. The transla- tion vector field b and the rotation vector fields fi determined by the connec- tion are defined by:

b = t ; Vx, f~ = / c V e ~ ,

where C is an infinitesimally small closed loop and boundary of a 2-dimensional submanifold S of M with the same induced connection F. The sense of traversing the loop is chosen such that the enclosed submanifold is to the left.

On the basis of the connection one forms w~ a foliation of the manifold (M, F) can be realised and choosing �89 - 1) pairs of them will yield submanifolds containing the desired submanifold S. These integral invariants are intrinsic vectors of the submanifold (S, F) and also of the manifold (M, F). Using Stokes' theorem the translation and rotation vector fields can be expressed as [5]:

b = s /i = s X?~ej.

254

At branching points the translation and rotation vector fields satisfy the following superposition principles (conservation laws):

One can conceive the latter principles just as the integral geometric version of the Bianchi identities [5,4].

Now let us demonstrate that an integral geometric operation suffices to detect certain types of singularity sets. In figure 1 the length of the trans-

vr for a discretised input image L0 on a lation vector field b with x = Ivr

two-dimensional Euclidean space E 2 is computed by means of linear scale- space theory [1]. The set of non-isolated singularities will instantaneously disappear upon linear scaling, but the ridges and ruts, and other type of dis- continuities seem to be nicely detected. However, the detection of the true ridges and ruts requires as mentioned in the previous section a multi-local geometric operation or an integral geometric operation with respect to the unit normal field of the set of flowlines along the isophotes. Performing a dif- ference operation in a distributional sense (integral invariant manner) across the ridges and ruts along the isophotes of the unit normal field of the set of fiowlines suffices to highlight the ridges and ruts.

Fig. 1. Left frame: a 256 • 256 pixel-resolution discrete input image L0(x, y) = e-SY2,x < O, Lo(x,y) = e-5(x+Y)2,x _> 0, y < 0 and Lo(x,y) = e-5(~-Y)2,x > 0, y ~ 0. Right frame: the Euclidean length of the translation vector Ibl for a linearly scaled version of that image.

255

3 Dynamic Scale-Space Theories

In subsection 3.1 dynamic scale-space theories for smoothing the input image are derived by determining the Beltrami-Laplace operator consistent with a particular choice of an image induced connection and metric. In subsection 3.2 dynamic scale-space theories for smoothing the image formation are derived by coupling the smoothing of the image formation to itself.

3.1 I n p u t Images

First let us denote with label re f and ind aspects related to the geometry of the image domain (the camera system) and the induced geometry by the input image, respectively. Normally the connection on the image plane is chosen flat and non-twisted causing the labelling of the connection by re f to coincide with a flat metric or a standard inner product on Euclidean space. Assume that dynamic scale-space theories for the input image can be based on the following conservation law for total grey-value on a region #ind of the image domain Mind with boundary OPind:

L~ 0L

where s is the scale parameter, O'~#ind is a normal field, Vind is the induced covariant derivative constructed on the basis of the induced connection ~2ind, "~ind is an induced metric tensor, q(#ind) and ~?(O#ind) are volume measures on the interior 0 Pind and the boundary O#ind , respectively. Using the divergence theorem and reflective boundary conditions these dynamic scale-spaces are governed by the following Cauchy problem:

OL ~ (~'---~ = ~/ind V ind,c~V ind,flL, (1)

OL - - 0 o n O~tin d X k s , nind �9 on~tind, ( 2 )

~r~ind L(x, O) = Lo(x), (3)

in which the first equation represents the scaling operation, the second equa- tion a reflective boundary condition ensuring the conservation of total grey- value and the third equation states the initial condition, and where ~8 the possible scale-range.

Requiring the dynamic scale-spaces for the input image to be invariant under spatially homogeneous grey-value transformations and applying a sim- ilar conservation principle as above for Euclidean geometries induced by the input image, the divergence theorem and a variational principle, the dynamic scale-spaces can be shown to be governed by the following Cauchy problem

256

[2]:

OL ~ 0--~ = --~/ref Vref,c~ L A i n d X f l , (4)

OL -- 0 on OPind X ~r~s, n in d e on/~ind, (5)

Onind L(x, 0) = L0(x), (6)

with X a position vector in Euclidean space describing a submanifold that is constructed on the basis of the set of isophotes and flowlines and with Ai~d the induced Beltrami-Laplace operator given by:

Z~in d --~ --"f~nfld ~ ind,a V ind,f~. (7)

In solving the above Cauehy problems for the above metric and connection image induced scale-space paradigms one is mainly interested in finding the corresponding Green's functions. For exact approximative Green's functions for above nonlinear Cauchy problems the reader is referred to [7]. There exist various methods to arrive at sensible connections and metrics. One of them is solving Euler-Lagrange equations related to some functional such as the Hilbert-Einstein action [3,4,8]. Using the found connection one can scale the input image or image formation consistently. Note that, for example the linear scale-space theory also has a Lagrangian formulation [2].

Let's conclude giving two examples of the above dynamic scale-space the- ories.

Example 11. The smoothing of a grey-valued input image on two-dimensional Euclidean space E 2 is normally steered by the image gradient through the conductivity tensor [9]. Alternatively, one could steer the heat capacity on the basis of the image gradient. Choosing a flat connection and a metric equal to

"/ = exp ( - ~ ) d x P @ dx p,

one obtains, upon substitution these choices into equation (1) and (3) and assuming the image domain not to be bounded, as Cauchy problem for the input image:

/ LiLi'~ o8 exp ZX sL, L(.,0) Lo(.),

where )~ a contrast parameter (see figure 2).

Example 12. Instead of steering the smoothing of a grey-valued input image on two-dimensional Euclidean space directly as in the above example one can

257

Fig. 2. Left frame: an input image of 256 • 256 pixel-resolution with dynamic range between 10 and 250 intensity units. Right frame: the nonlinear smoothing of the left input image according to the equations in example (11) (obtained by means of an explicit scheme [9]) with contrast parameter A = 5, time step 0.20 and after 200 iteration steps.

also prefer to control the smoothing indirectly choosing the connection and the metric as follows:

( _ ~ ) i : O, ~[ : 7]ref , W 1 = - exp L~ dv, w 2 = O, wj

where v and w are the coordinates with respect to the orthonormal frame field to the isophotes and substituting these choices into equations (4), (6) and (7). The dynamic scale-spaces of the input image are then governed by the following Cauchy problem:

OL ( L ~ ) 02L L ( - , 0 ) = L o ( . ) , Os - exp - ~ Ov--~,

which is just an alternative controlled Euclidean shortening flow to that con- sidered in [10,9].

3.2 I m a g e F o r m a t i o n

In anisotropic scale-space theories [9] one is concerned in retaining the discon- t inuity sets as much as possible under the smoothing. It is demonstrated in [2] that these theories have a nice geometric foundation. Instead of smoothing the input image by means of the inhomogeneous Euclidean group actions as proposed in the first reference in the sequel the smoothing of the image forma- tion is proposed as done in [2]. Because the image formation of a grey-valued

258

input image on a two-dimensional Euclidean space E 2 can be described as an inhomogeneous Euclidean group action induced by the input image it is natural to diffuse the translation vector field and the rotation vector fields (see section 2). As noted in geometric formulations of the anisotropic scale- space theories [2] the smoothing of the input image can be suppressed at discontinuity sets, such as ridges and ruts. Analogously, the smoothing of the image formation can be suppressed at the essential physicaI objects. Leaving the essential physical objects such as ridges and ruts as much as possible untouched the scaling of the translation vector field b and the rotation vector fields f~ should be defined as follows.

D e f i n i t i o n 13. The Euclidean dynamic scale-space theory for the transla- tion vector field b and the Frank vector fields fi related to a grey-valued input image on a two-dimensional Euclidean space E 2 are governed by the following Cauchy problems (system of partial fifferential equations):

0r (97" ~ i J v i ( O c t ) j , on ~ X ]R +

r = r on n

~/iJ(or162 on c9~ • +

with r the translation vector field or one of the rotation vector fields, Or a diffusion operator consistent with r and the metric connection !P, V) such that the components of the metric tensor are given by "/ij = ~i �9 Cj.

Note that in the case of the smoothing of the translation vector field b the metric becomes degenerate. Moreover, the flow j = - O ~ r can be taken in analogy with the anisotropic scale-space theories equal to:

j = - d V r d = exp ( - ( ~ r ,

with d similar to a structure tensor and c~r a contrast parameter. On the basis of the divergence formulation of these dynamic scale-space theories the evo- lution of the image formation in terms of the translation vector field and the rotation vector field can readily be computed [7,9]. Concluding these dynamic scale-space theories enable a multi-scale description of the image formation in terms of scaled measures of torsion and curvature. Note that similar dy- namic scale-space theories can be formulated, if the image formation should be invariant under the group of monotonic grey-value transformations.

4 D i s c u s s i o n

Modern geometry is demonstrated to be useful in describing the image for- mation of a grey-valued input image on a two-dimensional Euclidean space. The torsion and curvature of the image formation can be represented in terms

259

of differential and integral invariants that read out the inhomogeneous Eu- clidean group action caused by an image induced connection. In particular the translation vector fields and rotation vector fields appear to be useful to quantise the image formation near essential physical objects such as ridges, ruts and other types of singularities or discontinuities. Next on the basis of conservation laws and additional conditions dynamic scale-space theories are derived for the input image or the image formation

The question arises how to generalise these dynamic scale-space theories. If one requires the dynamic scale-space theories to be topological equivalent, i.e. the theory should not be affected by anamorphoses of the input image, then one can turn to dynamic scale-space theories taking only the landscape of ridges, ruts and the singularities as edges and vertices of a random lattice into account. Assuming the total grey-value within one cell enclosed by ridges, ruts and singularities still to be constant the average total grey-values at tained at the singularities can savely be redistributed over the lattice according to the the local connectivity structure of the landscape, i.e. the number of non-isolated singularities, ridges and ruts ending at a vertex, and the typical average total grey-values at the neighbouring singularities. One arrives at a true covariant or better topologically equivalent scale-space theory.

R e f e r e n c e s

1. ter Haar Romeny, B.M., ed.: Geometry-Driven Diffusion in Computer Vision. Dordrecht: Kluwer Academic Publishers, 1994.

2. Salden, A.H.: Dynamic Scale-Space Paradigms. PhD thesis, Utrecht University, The Netherlands, 1996.

3. Kadi, A., Edelen, D.: "A gauge theory of dislocations and disclinations," in Lecture Notes in Physics, vol. 174. Araki, H., Hepp, E.J.K., Kippenhahn, R., Weidenmiiller, H.A., Zittartz, J. eds. Berlin: Springer-Verlag, 1983.

4. Kleinert, H.: Gauge Fields in Condensed Matter, vol. 1-2. Singapore: World Scientific Publishing CO. Pte. Ltd., 1989.

5. Caftan, E.: Sur les vari~tds h connexion aJ~fine et Ia thdorie de la relativitd gdndralisde. Gauthiers-Villars, 1955.

6. Koenderink, J.J., van Doorn, A.J.: "Local features of smooth shapes: Ridges and courses," in Proceedings SPIE Geometric Methods in Computer Vision II, vol. 2031, San Diego, CA, (1993) 2-13

7. Eidel'Man, S.: Parabolic Systems. Amsterdam, Groningen: North-Holland pub- lishing Company and Wolters-Noordhoff Publishing, 1962.

8. Hehl, F., Dermott McCrea, J., Mielke, E.: WeyI Space-times, The Dilation Current, And Creation Of Gravitating Mass By Symmetry Breaking. Frankfurt: Verlag Peter Lang (1988) 241-311

9. Weickert, J.: Anisotropic Diffusion in Image Processing. PhD thesis, Dept. of Mathematics, University of Kaiserslautern, Germany, 1996.

10. Alvarez, L., Lions, P.-L., and Morel, J.-M.: "Image selective smoothing and edge detection by nonlinear diffusion. II," SIAM J. Num. Anal. 29 (1992) 845-866.