www.nr.no

INF 3300, INF4300Deformable contoursAn introduction

Lars Aurdal,Norsk Regnesentral,

8th August 2005

www.nr.no

Practical information

1. All foils will be made available on the course web site.

2. The foils define the course requisites.

3. Exercises will be introduced as we go along.

www.nr.no

Plan

1. Motivation, why deformable contours?

2. Snakes.

3. Some more advanced aspects of deformable contours.

4. Applications of deformable contours.

www.nr.no

Motivation

1. Classical assumption: Digital images will show real world objects as well defined regions with unique gray levels and a clear border against a uniform background.a. This assumption can hold true in certain cases, but

typically it does not.

2. Real world objects seen in digital images might very well blend with the background and many parts of the objects might be more or less occluded.

www.nr.no

Motivation

Beware of extreme case of blending and occlusion

www.nr.no

Motivation

1. Classical assumption: The image contains all the information necessary in order to produce a good segmentation.a. Again, this might be so in certain situations, but this

is more of an exception than the rule.

2. In real world images we will typically need higher level information about the objects we are trying to segments:a. Sizeb. Shapec. Position

www.nr.no

Motivation

1. We have already seen many examples where such knowledge is used:a. Thesholding: We assume knowledge of

graylevels….b. Adaptive thesholding: We assume knowledge of the

size of the objects we are interested in…c. Hough transform: We assume certain shapes…

www.nr.no

Motivation

1. The methods we will look at today are just another example of segmentation methods using models external to the image in order to obtain the best possible segmentation.

2. Typical segmentation problems solved with this kind of image are for instance ultrasound cardiac images.a. Much noise and blurred edgesb. Much knowledge about shape etc.

www.nr.no

Motivation

Left ventricle of the heart, apical view.© GE medical

www.nr.no

The initial idea: Snakes

1. Snakes are model-based methods for localization and tracking of image structures.

2. The snake is defined as an energy minimizing contour (often splines).

3. The energy of the snake depends on its shape and location within the image.

4. Snakes are attracted to image boundaries through forces.

www.nr.no

The initial idea: Snakes

1. The approach is iterative:a. The user draws an initial approximate contour.b. A dynamic simulation is started.c. The contour is deformed until it reaches equilibrium.

2. Snakes depend on:a. Interaction with user.b. Interaction with high level description.c. Interaction with image data adjacent in space and

time.

www.nr.no

The initial idea: Snakes

1. Different types of forces will affect the snake:a. Internal forces.

1) Tension forces that make the snake act like an assembly of springs.

2) Rigidity forces that make the snake act like a thin wire that resists bending.

2. Image forces

3. Constraint forces.a. User supplied forces or forces that come from higher

level image understanding processes.

www.nr.no

The initial idea: Snakes

1. The contour is represented as:

2. This is a parametric representation of the contour.

3. The vector describing the position of every point on the contour makes one pass over the entire contour as s varies from its minimum to its maximum value. Typically s is normalized:

www.nr.no

The initial idea: Snakes

1. What is a parametric contour?

2. Let x(s)=cos(2π s) and y(s)=sin(2π s). Furthermore s∈ [0,1].

3. Then v(s) describes a circle as s varies from 0 to 1.

www.nr.no

The initial idea: Snakes

1. The shape of the contour v(s) is dictated by a functional:

2. This functional can be seen as representing the energy along the contour.

3. The final position of the contour will correspond to a minimum of this energy function.

www.nr.no

The initial idea: Snakes

1. The first term of the functional can be viewed as a representation of the internal deformation energy:

www.nr.no

The initial idea: Snakes

1. The first term:

2. imposes tension, that is, we want a curve as short as possible.

www.nr.no

The initial idea: Snakes

1. The second term:

2. imposes rigidity, changes in direction should be smooth.

www.nr.no

The initial idea: Snakes

1. The weight functions w1(s) and w2(s) are used to modify the weights given to the two terms.

2. Why do you think it is a smart idea to make these a function of s? Can you think of cases where this might be a nice feature.

www.nr.no

The initial idea: Snakes

1. The second term of the functional introduces an external energy term into the expression for the functional:

www.nr.no

The initial idea: Snakes

1. In this expression, P(x,y) denotes a scalar potential function defined on the image plane.

2. We obviously choose potential functions P(x,y) whose minima coincide with intensity extremas, edges or other interesting features of the image.

www.nr.no

The initial idea: Snakes

1. The most common way of defining P(x,y) is the following:

2. What does this expression mean in terms of the final position of the snake.

www.nr.no

The initial idea: Snakes

1. A variation of this can be obtained by letting:

2. What would be the use for this kind of definition?

www.nr.no

The initial idea: Snakes

1. The contour is initiated somewhere in the image.

2. The contour is represented as a number of control points, the control points can typically be interconnected using splines.

3. The control points are moved iteratively in a direction determined by forces derived from the internal energy of the contour and the external energy affecting the contour.

4. The final resting position of the snake is a position where a balance is found between the internal and the external forces.

5. In this position the snake will have certain geometric properties related to length and smoothness, it will also (hopefully) indicate salient edges in the image.

www.nr.no

The initial idea: Snakes

1. This was the initial idea of Kass and Witkin in their very important paper: Snakes: Active contour models, Michael Kass, Andrew Witkin and DemetriTerzopoulos, International Journal of Computer Vision, pages 321-331, 1988.

www.nr.no

Snakes: problems and pitfalls

1. The basic snake formulation suffers from many drawbacks.a. Initialization has to be fairly good, if the snake is

initiated to far from the contour then it will not converge.

b. Even with a good initialization, we depend on correctly setting w1 and w2.

c. Topological problems.d. Only 2D.

www.nr.no

Snakes: problems and pitfalls

1. The choice of w1 and w2 can only be resolved by experimentation.

2. In addition, we require that every instant of the problem is sufficiently similar to other instances so that the same w1 and w2 can be used.

3. In practical life, this is often not a very serious problem! We typically want to apply our algorithms to the same type of images over and over.

www.nr.no

Snakes: problems and pitfalls

1. The problem of topology is the following: The classical snake is connected, it cannot split to adopt to objects with, for instance, holes.

2. This can be a problem in many situations.3. This problem has been addressed by many researchers and

solutions exist. See for instance: Topology Adaptive Deformable Surfaces for Medical Image Volume Segmentation, Tim McInerney and Demetri Terzopoulos, IEEE Transactions on Medical Imaging, vol. 18, no. 10, 1999.

4. This work also deals with an extension to 3D.5. In practical life, the topological adaptability of snakes is not

necessarily a big issue! The most frequent applications of snakes use non-topologically adaptable snakes.

www.nr.no

Snakes: problems and pitfalls

1. The final problem, that of the initialization is possibly the most serious of these problems.

2. To make a really good initialization we need to have a very good estimate of the solution before starting the iterative process of adapting the snake.

3. So we can find a good solution if we already know the solution. That is not very interesting of course.

www.nr.no

Snakes: problems and pitfalls

1. The problem stems from the short range of the external forces.

2. The gradient (or smoothed gradient) vector field used to move the snake will have significant values only in the vicinity of the salient edges.

3. This basically forces us to initialize the snake very close to the target contour.

4. This problem is also known as the capture range problem.

www.nr.no

Snakes: problems and pitfalls

1. Many authors have suggested different methods for increasing the capture range of the external gradient vector field.

2. We will look at one elegant method suggested by Xu and Prince in their article: Snakes, Shapes and Gradient Vector Flow, Chenyang Xu and Jerry Prince, IEEE Transactions on Image Processing, vol. 7, no. 3, pages 359-369, 1998.

www.nr.no

Snakes: examples

1. This is a synthetic binary image:

www.nr.no

Snakes: examples

1. It would be nice to be able to initialize the snake as shown below and have it converge, but will it?

www.nr.no

Snakes: examples

1. Matlab example!

2. Xu and Prince example (from http://iacl.ece.jhu.edu/projects/gvf/).

www.nr.no

Snakes: GVF solution

1. Xu and Prince observed that smoothing will increase the capture range, but will not provide convergence within concavities.

2. Other methods, for instance those based on distance maps are even better at increasing capture range, but the concavity problem remains the same.

3. Xu and Princes solution is based on diffusing the gradient information.

www.nr.no

Snakes: GVF solution

1. Xu and Prince defines the gradient vector flow (GVF) field to be the vector field that is the equilibrium solution to the following diffusion equation:

www.nr.no

Snakes: GVF solution

1. You need to choose g() and h() intelligently:

2. One possibility is to let g be a constant:

3. and h be equal to the square of the magnitude of the gradient vector field:

www.nr.no

Snakes: GVF solution

1. Inspect the gradient vector field after such diffusion:

From: http://iacl.ece.jhu.edu/projects/gvf/

www.nr.no

Snakes: applications

1. Segmenting poorly defined contours (as for instance in the case of ultrasound images of the heart).

2. Tracking of moving objects.a. Let the equilibrium solution from one frame be the

starting point in the next frame.