Image Matching and Retrieval by Repetitive PatternsPetr Doubek, Jiri Matas, Michal Perdoch and Ondrej Chum

Center for Machine Perception, Czech Technical University in Prague, Czech Republic

Detection of repetitive patterns in images is a well-established computer vision problem. However, the detected patterns are rarely used in any application. A method for representing a lattice or line pattern by shift-invariant descriptor of the repeating tile is presented. The descriptor respects the inherent shift ambiguity of the tile definition and is robust to viewpoint change. Repetitive structure matching is demonstrated in a retrieval experiment where images of buildings are retrieved solely by repetitive patterns.

Motivation• Multiple occurrences of a local patch pose a problem to one-to-

one matching algorithms (local matches are ambiguous)• The presence of repeated local patches is in most cases non-

accidental and therefore very distinctive• Regular and even non-regular repetitive patterns give a rise to

geometric constraints

Repetitive Pattern and Lattice Detection

Original image Detected lattice Detected tiles Zero-phased tiles

Shift-Invariant Tile Representation• Each repeated pattern is represented by an “average” appearance of a tile – a mean tile(a more complex representation possible)• After discovering lattice of a repeated pattern an intrinsic shift ambiguity remains• We propose two shift-invariant representations:

• the magnitude of Fourier coefficients of a tile• “zero-phase” normalization: tile shifted so that phase of the first harmonic equals zero

Outline of the AlgorithmDetection1. Detect repeated elements, find the lattice of the repeated

pattern, rectify the lattice and calculate the mean tile2. Compute shift invariant tile descriptorsImage Matching and Retrieval3. For each pair score with the most similar patterns

1. Detection of repeated elements. In our implementation affine covariant regions (MSERs and Hessian Affine) described by SIFT

2. Agglomerative clustering of SIFTs. Each cluster hypothesise a repeated pattern

3. For each element in the repeated pattern, find spatial nearest neighbours in each of the spatial sectors

4. Find dominant vanishing points by Hough transform and form a 2D lattice

5. Rectify lattice and divide pattern into tiles

A cluster of repeated elements

Nearest neighbours and neighbourhood sectors

A lattice from two vanishing points (corresponding to red

and green directions)

• For a pair of images i,j with sets of detected repeated patterns Ci and Cj similarity sk,l of two patterns k,l is computed as

where and are shift invariant tile descriptions and are pairs of peaks in RGB colour histograms.

Repeated Patterns Similarity

Image Retrieval by Repetitive Patterns

Dataset Query Top three best matches .

Experiment 2• Detection and matching of repeated patterns tested on image retrieval• Two publicly available datasets PSU-NRT(subset) and Pankrac+Marseille (

http://cmp.felk.cvut.cz/data/repetitive)• Top three matches for some of the queries

Experiment 1• Performance of shift invariant representation• Ground truth for each query Gi = set of images of

building i, was manually marked

• Tested on two datasets with ~230 images in image retrieval of about 50 buildings

Conclusions• Image retrieval can benefit from repeated patterns if

they are detected and handled properly• Proposed approach is able to detect 1D and 2D

lattices under affine transformation• Shift invariant descriptors addresses tile ambiguity• We have shown retrieval based solely on repeated

patterns, however it can be combined with standard bag-of-words retrieval approaches

References• T. Tuytelaars, A. Turina, and L. Van Gool, “Non-combinatorial

detection of regular repetitions under perspective skew”, PAMI, vol.25, no.4, pp. 418-432, April 2003

• P. Doubek, J. Matas, M. Perdoch and O. Chum, “Detection of 2D lattice patterns of repetitive elements and their use for image retrieval”, technical report, CTU-CMP-2009-16, 2009

• T. K. Leung and J. Malik, “Detecting, localizing and grouping repeated scene elements from an image”, ECCV, 1996, pp. 546-555

The authors were supported by Czech Science Foundation Project 102/07/1317 and by EC project FP7-ICT-247022 MASH.

Example: repeated patterns of two images and their similarity

pik = (pik;1;pik;2), p

jl = (pjl;1;p

jl;2)

dik djl

Abstract

0.56 0.37 0.07 0.08 0.06 0.42

0.00 0.01 0.00 0.00 0.03 0.02

10x10 43px 12x10 27px 6x9 39px 2x6 53px 16x1 15px 7x11 63px

12x10 57px

17x4 73px