R. Wilson et al. (Eds.): CAIP 2013, Part II, LNCS 8048, pp. 201–208, 2013. © Springer-Verlag Berlin Heidelberg 2013

Multi-scale Image Segmentation Using MSER

Il-Seok Oh1, Jinseon Lee2, and Aditi Majumder3

1 Division of Computer Science and Engineering/Center for Advanced Image and Information Technology, Chonbuk National University, South Korea

isoh@chonbuk.ac.kr 2 Department of Games and Contents, Woosuk University, South Korea

jslee@woosuk.ac.kr 3 Department of Computer Science, University of California, Irvine, CA

majumder@ics.uci.edu

Abstract. Recently several research works propose image segmentation algo-rithms using MSER. However they aim at segmenting out specific regions cor-responding to user-defined objects. This paper proposes a novel algorithm based on MSER which segments natural images without user intervention and captures multi-scale structure. The algorithm collects MSERs and then parti-tions whole image plane by redrawing them in specific order. To denoise and smooth the region boundaries, hierarchical morphological operations are devel-oped. To illustrate effectiveness of the algorithm’s multi-scale structure, effects of various types of LOD control are shown for image stylization.

Keywords: Multi-scale structure, image segmentation, image stylization.

1 Introduction

Multi-scale image segmentation aims at producing a hierarchical tree-structured parti-tioning of image plane to analyze scene in a series of scales [1]. This output is useful for LOD (level-of-detail) control in various tasks such as image stylization and medical diagnosis. For example, DeCarlo stylized images by drawing fine details at user-specified salient regions while drawing other parts coarsely [2]. Conventional algorithms construct a multi-scale representation by applying Gaussian convolutions in well-controlled multiple levels. Then a linking process is used which finds corres-ponding blobs in adjacent scales [1, 3, 4].

Recently several researchers propose image segmentation algorithms inspired by MSER (maximally stable extremal region) technique that produces blobs. The blobs are highly featured regions in the sense that they are stable and salient, and have mul-ti-scale structure [5]. When compared to the algorithms which rely on intensity ex-trema, MSER is more stable because it stems from stability extrema (as explained in Section 2). This stability of MSER has been shown to be superior to other techniques in performance benchmarking of local features [6].

Donoser proposed a color image segmentation algorithm which requires an user to indicate an ROI (region-of-interest) and uses it to order RGB values [7]. Gui used

202 I.-S. Oh, J. Lee, and

MSER as initial segmentapassed the results to spectration from natural images isever the algorithms aim adefined rules. AdditionallyMSER tree.

In this paper, we proposeMSER. The approach can accomplishes the segmentathe image plane in an apprwhole image. To denoise ancal operations and a desirabness of the algorithm’s mulover natural images are sho

The contributions of this

• Extending blob detection• Presenting tree traversal • Validating the effectiven

controls for image styliza

2 Algorithms

An ER (extremal region) isintensity values than a thresholds ti and tj. For an ER athe change in area Ψ(ti) instability, i.e., minimum Ψ. in Fig. 1, ER at ti is probabl

Ψ

Fig. 1. 1-D illustration of MSE

The MSERs are nested andusual to perform MSER de

A. Majumder

ation tool for SAR (synthetic aperture radar) images al clustering process [8]. One of recent trends in text det to segment candidate text regions using MSER [9]. Ho

at detecting only specific object regions based on uy they do not address using the multi-scale structure

e a new multi-scale image segmentation approach basedsegment natural images without any user intervention

ation by collecting MSERs and then rearranging them oropriate order that would generate desired segmentationnd smooth the region boundaries, hierarchical morpholoble sequence of them are developed. To illustrate effectilti-scale structure, effects of various types of LOD con

own for image stylization. s paper are as follows:

n functionality of MSER to segmentation of natural imagalgorithms for image partitioning and boundary smooth

ness of the multi-scale structure by showing various Lation

defined as a connected component of pixels having higeshold [5]. Fig. 1 shows a 1-D version of ER at two that ti, we can define the stability as an inverse of the ration Equation (1). The MSER is an ER with local maxim

We call this property as stability extremum. For example to be MSER while one at tj is not.

i

iii area

areaareat Δ+Δ− −

=Ψ )(

ER+ and MSER- (from left to right: ER, MSER, component tr

d represented by a component tree shown in Fig. 1. Ietection twice, one for bright MSER+ and another one

and tec-ow-ser-

e of

d on n. It onto n of ogi-ive-

ntrol

ges hing LOD

gher hre-o of

mum mple,

(1)

ree)

It is for

Multi-scale Image Segmentation Using MSER 203

dark MSER-, pointed by T+ and T- in the component tree, respectively. A collision between MSER+ and MSER- may happen, as exemplified by a3 and b2 in Fig. 1.

The nested structure provides multi-scale structure because we can choose any lev-el of details by pruning the tree as we need. This property of MSER is the most po-werful one from the perspective of our proposed image segmentation technique. A linear O(n) algorithm to construct the component tree has been developed in [10] where n is the number of pixels. So image segmentation based on MSER can be ex-pected to be very fast. We propose such a technique in the following section.

Fig. 2 shows a natural image taken from Berkeley database [11] and two segmenta-tion results by MSER+ and MSER- trees. The white area indicates being not covered by MSERs.

Fig. 2. Natural image and segmentation results (from left to right: original, MSER+, MSER-)

2.1 Image Segmentation Using MSER

Our method is based on post-order traversal of the component tree that partitions the image. The pseudo code is as follows.

Algorithm 1: Image segmentation by post-order traversal of component tree 1. function postorder(t) 2. if(t is leaf node) partition(I, t) 3. else 4. for(each of children, c of t) postorder(c); 5. partition(I, t)

In order to segment using MSER+, function call, postorder(T+) is used. For

MSER-, postorder(T-) is used. The function partition(I,t) labels the pixels of image I belonging to the node t. Labeling is done only when the pixel has not been labeled before so that child node has a priority over parent node. For the residual parts not covered by MSERs (white area in middle and right images of Fig. 2), assigning spe-cial label completes the image segmentation.

Inspection of the segmented images in Fig. 2 shows that MSER+ or MSER- alone cannot segment whole image. This is while MSER+ captures the regions of bright ob-jects, MSER- captures the dark ones. Hence, both the MSER+ and MSER- component trees should be considered to get a complete image segmentation. To achieve this, we present a technique to perform postorder(T+) and postorder(T-) in the following section.

204 I.-S. Oh, J. Lee, and A. Majumder

2.2 Core Node-First Traversal

For this, we first define a merger node and core node. A merger node is a node with more than or equal to two children. A core node is a node in the component tree that is be-tween a leaf node and the first merger node encountered while moving upwards from the leaf node. In Fig. 1, a6 is merger while others are core nodes. The regions of core nodes can be regarded as photometrically salient since they have distinctive intensities from surrounding regions and are called core regions. To find the core regions, we present another traversal algorithm which endows priority to the core nodes.

Algorithm 2: Image segmentation by core-first traversal 1. core_first(T+) 2. core_first(T-) 3. postorder(T+) 4. postorder(T-) 5. function core_first(t) 6. if(t is leaf node) 7. repeat 8. partition(I, t) 9. prune t // make it dead 10. t=parent(t) 11. until (t is not merger node) 12. else 13. for(each of children, c of t) core_first(c)

Algorithm 2 applies the function core_first() for each of two trees successively,

MSER+ and MSER-. Successive applications of postorder() for two trees complete the segmentation process. The pruned dead node is excluded in the process of post-order(). Fig. 3 shows the core regions thus deciphered from MSER+ and MSER-. For this example image, no collision has occurred between them. Observation over lots of

Fig. 3. Core-first segmentation (from left to right: cores from MSER+, cores from MSER-, both, results of Algorithm 2 (Colors encoding node’s depth in tree, red (depth 2), green(3), blue(4), ...))

images revealed that no or ocollision zones is determinefirst-occupy rule. An elaboredge information as [3].

3 Hierarchical Mo

Like any other segmentatiosmoothing. We consider mthe segmentation result is nchild-parent relationship, spropose the following hiera

Algorithm 3: Child-first dil1. function postorder_2. if(t is leaf node)3. else 4. for(each of chi5. dilation(t) 6. function dilation(t)7. for(each pixel, p8. for(each pixel9. if(q is ance

Fig. 4. Hierarchical morpholochildren’s label)

Fig. 4 explains the abovthe parent of node 5 and noturing element. The net effeancestor regions about one

We can easily change Atraversal is changed from po

Multi-scale Image Segmentation Using MSER

only a slight collision happens. Currently occupation of ed arbitrarily between MSER+ and MSER- by first-comration can be made in the future, for example by exploit

orphology

on methods, our method also needs denoising and boundmorphological operations for this purpose. However sinot binary but encoded in a hierarchical manner with

standard operations are not readily applicable. Hence, archical morphology algorithm:

ation _dil(t) dilation(t)

ildren, c of t) postorder_dil(c)

) p of node t) l, q in S[p]) // S:structuring element

estor of p) I[q]=I[p]

ogy (The label of underlined pixels should be changed into t

e dilation operation on a hierarchical map where node ode 7 is the parent of node 6. This example uses 3*3 strfect of this dilation is the expansion of children region ipixel wide.

Algorithm 3 into a parent-first version as follows. Here ost-order to pre-order.

205

f the me-ting

dary ince the we

their

6 is ruc-into

the

206 I.-S. Oh, J. Lee, and A. Majumder

Algorithm 4: Parent-first dilation 1. function preorder_dil(t) 2. if(t is leaf node) dilation1( t) 3. else 4. dilation1(t) 5. for(each of children, c of t) preorder_dil(c) 6. function dilation1(t) 7. for(each pixel, p of node t) 8. for(each pixel, q in S[p]) 9. if(p is ancestor of q) I[q]=I[p]

Note that in function dilation1(), the condition of if statement has been changed by

reversing roles of p and q. Now we may apply various combinations of C (Child-first) and P (Parent-first) such as C, CC, CP, CPC. Fig. 5 shows effects of CPC scheme. Experiments with many natural images showed that CPC scheme produced good re-sults. So in the remainder of this paper, we will use CPC scheme.

Fig. 5. Results before (left) and after (right) hierarchical morphology

4 Image Tooning Using Multi-scale Structure

The multi-scale structure embedded in the component tree can be used in a variety of purposes. In this section, we use this to present effects of image tooning [12] that uses a simple technique of assigning a region its average color. We illustrate this in Figure 6. The leftmost image is the original image from Berkeley database. Pruning nodes of surfer’s body resulted in second image from left in which details disappeared. Right two images were produced by pruning nodes for sky and big waves.

Multi-scale Image Segmentation Using MSER 207

Fig. 6. Styling using multi-scale structures (left: original, the remainings: stylized)

Fig. 7 illustrates the use of the multi-scale structure for a stylization in which core regions and non-core regions are treated differently. Three right images of Fig. 7 are ones where boundaries are emphasized by black lines. The fourth from left was gen-erated by decreasing intensities of non-core regions by multiplying by 0.8 while in-creasing the intensities of cores by multiplying by 1.2. To accomplish this RGB is first converted into CIELab color space and the multiplication is done for L (lumin-ance) component. Then the image is converted back to RGB. The effect is contrast enhancement between core and non-core. Rightmost is of reverse effect by increasing intensities of non-core regions while decreasing the intensities of cores.

Fig. 7. Contrast enhancing of core regions (from left to right: original, core regions (non-core in white), core 1.0 and non-core 1.0, core 1.2 and non-core 0.8, core 0.8 and non-core 1.2)

5 Concluding Remarks

The paper described an approach to extend MSER’s blob detection functionality to the image segmentation. Appropriate traversal algorithms of component trees were presented and evaluated. The results revealed both the possibilities and limitations. We validated the effectiveness of multi-scale structure by showing various LOD con-trol for image tooning. The paper emphasized that this was achieved without multi-level Gaussian smoothing.

In spite of good properties of MSER, some limitations still exist. We summarize them along with important future directions.

• The proposed algorithm has no process for optimizing boundary localization accuracy and smoothness and no use of color information. New formulation of MSER extrac-tion by embedding edge-guided boundary tuning [3], smoothness prior, use of color [13], and contrast enhancement [14] would make MSER to be utilized very success-fully as a true image segmentor. The revision should be done in the realm of retaining MSER’s nature of seeking stability extrema and multi-scale structure.

208 I.-S. Oh, J. Lee, and A. Majumder

• The proposed algorithm is expected to be advantageous in video segmentation due to its nested structure and stability. Constructing and segmenting 3-D volumes ((x,y) for position and t for time) using MSERs is an important future work. For this purpose, faster and more robust MSER tracking algorithm proposed in [15] can be used.

References

1. Koenderink, J.J.: The structure of images. Biological Cybernetics 50, 363–370 (1984) 2. DeCarlo, D., Santella, A.: Stylization and abstraction of photographs. SIGGRAPH,

769–776 (2002) 3. Chen, J., et al.: Edge-guided multiscale segmentation of satellite multispectral imagery.

IEEE Tr. Geoscience and Remote Sensing 50(11), 4513–4520 (2012) 4. Petrovic, A., et al.: Multiresolution segmentation of natural images: from linear to nonli-

near scale-space representations. IEEE Tr. Image Processing 13(8), 1104–1114 (2004) 5. Matas, J., et al.: Robust wide baseline stereo from maximally stable extremal regions. In:

British Machine Vision Conference, pp. 384–396 (2002) 6. Mikolajczyk, K., et al.: A comparison of affine region detectors. International Journal of

Computer Vision 65(1/2), 43–72 (2005) 7. Donoser, M., Bischof, H., Wiltsche, M.: Color blob segmentation by MSER analysis. In:

IEEE International Conference on Image Processing, pp. 757–760 (2006) 8. Gui, Y., Zhang, X., Shang, Y.: SAR image segmentation using MSER and improved spec-

tral clustering. EURASIP Journal on Advances in Signal Processing (2012) 9. Shi, C., et al.: Scene text detection using graph model built upon maximally stable extrem-

al regions. Pattern Recognition Letters 34, 107–116 (2013) 10. Nistér, D., Stewénius, H.: Linear time maximally stable extremal regions. In: Forsyth, D.,

Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 183–196. Sprin-ger, Heidelberg (2008)

11. Martin, D., et al.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV (2001)

12. Kyprianidis, J.E., et al.: State of the art: a taxonomy of artistic stylization techniques for images and video. IEEE Tr. Visualization and Computer Graphics (2012)

13. Murphy-Chutorian, E., Trivedi, M.: N-tree disjoint-set forests for maximally stable extremal regions. In: British Machine Vision Conference, pp. 739–748 (2006)

14. Majumder, A., Irani, S.: Perception-based contrast enhancement of images. ACM Tr. On Applied Perception 4(3), 1–22 (2007)

15. Donoser, M., Bischof, H.: Efficient maximally stable extremal region (MSER) tracking. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 553–560 (2006)