Detection, Rectification and Segmentation of Coplanar Repeated Patterns
-
Upload
james-pritts -
Category
Science
-
view
64 -
download
1
Transcript of Detection, Rectification and Segmentation of Coplanar Repeated Patterns
Detection, Rectification and Segmentation of Co-planar Repeated
PatternsJames Pritts
Ondrej Chum and Jiri Matas
Center for Machine Perception (CMP)Czech Technical University in Prague
Faculty of Electrical Engineering Department of Cybernetics
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Introduction Repetitive patterns are ubiquitous in images Unless considered, they usually decrease vision algorithm
performance Seek a model-based approach to precisely locate and
segment general co-planar repeats
2/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
GOAL: Create a short-list of non-random matches to query image
Because of “burstiness”, repeated elements are over-counted:
High frequency words from repeats skew scoring
Co-occurring features are not independent
Image form H. Jegou and Matthijs Douze, On the burstiness of visual elements. In CVPR, 2009.
Problems with Repetitions: Image Retrieval
Query
Match??3/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Problems with Repetitions: Stereo Matching
Cannot disambiguate tentative correspondences F-estimate invalid
Epipolar constraint provides only weak spatial verification (even with good F)
4/25
mismatchedmismatched
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Prior Work Detecting repeats is well studied
Rectification is nearly universal (vanishing lines)
Exploit some constraint that is valid in rectified space
Lattice Schaffalitzky, F., Img. Vis. Comp.
2000 Lattice, Axial Symmetry
Wu et al, CVPR 2011 Symmetry
Hong et al, IJCV 2004 Congruency
Liebowitz et al, CVPR 1998 Aiger et al, Comp. Graph. Forum
20125/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
The State of the Art TILT: Zhang et al., IJCV 2012.
Find homography minimizing image rank
Manual cueing of pattern required
Fails with significant perspective warp, occlusions or if repeats are sparse
Aiger et al, Comp. Graph. Forum 2012 Joint maximization of
congruent line segments has no convergence guarantee
Systems of rational equations sampled by Hough transform (slow)
6/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Problem Formulation
Task: Segment imaged co-planar repeats with pixel level accuracy Some scene element repeats on a plane Need not have any regularity or be densely sampled Work without image structure modulo the repeat
- A common assumption is the existence of vanishing lines in the image that can be used to rectify the scene plane
Work when repeats cover only a small part of the image Fully automated: no cueing is required Can segment pattern to pixel level accuracy
Assumptions Repeated scene elements are coplanar Scene elements can be mapped to each other by rigid
transforms Imaged by perspective camera Scene element is repeated at least 3 times
7/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Proposed method A method for detection, precise alignment and segmentation of
general co-planar repeated patterns
8/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Proposed method A method for detection, precise alignment and segmentation of
general co-planar repeated patterns
8/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Proposed method A method for detection, precise alignment and segmentation of
general co-planar repeated patterns
8/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Intra-Image Feature Correspondence
Extremal regions (MSERs) detected for local representation of images
Local Affine Frames (LAFs) derived from extremal regions to concisely capture local geometry
Affine frames described by SIFTs
T(Ax)=AT(x)Affine covariance
11/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Feature Correspondence to Clusters Cluster: set of LAFs that are photometrically consistent.
12/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
From Clusters to Repeats: Rectification
Spatial verification of photometric clustering is needed Perspective and affine imaging does not preserve scale or
congruency Need general rectification method for rigidly transformed repeats
13/25
image rectified
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Rectification Stratum Translated and rotated co-
planar pattern
affinity similaritysimilarity w\scale ambiguity
14/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Rectification Stratum Translated and rotated co-
planar pattern
Translation: Affine rectification
affinity similaritysimilarity w\scale ambiguity
14/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Rectification Stratum Translated and rotated co-
planar pattern
Translation: Affine rectification
Rotation: Similarity rectification
affinity similaritysimilarity w\scale ambiguity
14/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Rectification Stratum Translated and rotated co-
planar pattern
Translation: Affine rectification
Reflection: Similarity with scale ambiguity along reflection axis
Rotation: Similarity rectification
affinity similaritysimilarity w\scale ambiguity
14/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Affine Rectification (Chum et al [3]) Assumption: repeated elements in real world are equiareal. Constraint: images of repeated elements should be equiareal after affine
rectification.
Source imageCoordinates and scales are known
Destination imageOnly scales are known (no positions)
H
Result: unit area ratio, but not necessarily equal angles and extent length ratios
18/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Similarity Rectification Assumption: repeated elements in real world have equal extent lengths. Constraint: images of repeated elements should have equal extent lengths. Result: Equal area ratios, relative extent lengths preserved, equal angles
Rotation Reflection
Imaged
Scene
2 LAFS needed 1 LAF needed
19/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Affinity removal with reflections
20/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Affinity removal with reflections
20/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Rectification Results
22/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Repeats to Motifs Repeat: photometrically consistent cluster of local affine frames
(LAFs) that satisfies scale constraint
Motif: is a collection of repeats that are spatially coherent
Instance: An occurrence of the motif in the pattern
Goal: Estimate a motif and set of transforms between
23/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Motif Estimation Open Problem: Formulate cost that balances model complexity and
motif cardinality
24/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Greedy Motif Construction
25/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Greedy Motif Construction
25/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Generative Model Generate the imaged pattern from the motif
motif
27/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Generative Model Generate the imaged pattern from the motif Estimate pattern and rectification from image
SIFTs extracted from image and clustered Rectification estimated from linear constraints Clusters verified against scale constraint to make repeats Geometric hashing of LAFS in rectified space to construct
motif
motif
27/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Generative Model Generate the imaged pattern from the motif Estimate pattern and rectification from image
SIFTs extracted from image and clustered Rectification estimated from linear constraints Clusters verified against scale constraint to make repeats Geometric hashing of LAFS in rectified space to construct
motif Refine pattern, rectification and lens distortion by minimizing
pattern re-projection error
motif
27/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Motif Construction
28/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Motif Construction
28/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Cows
30/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Cows
30/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Multiple motifs
34/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Multiple motifs
34/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Multiple motifs
34/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Future Work Seek join estimation of photometric clustering and rectification
Sequential estimation is error prone, especially for multiple planes
Failure modes
Infer rectification jointly from more constraints Broaden the class of images from which patterns can be
extracted Model complexity cost to principally estimate number of planes Formulate optimization problem for motif construction Integrate into image retrieval engine
37/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Conclusions Demonstrated effectiveness of new linear constraints
valid for nearly all man-made patterns effective in a fast RANSAC framework
Improved the state-of-the-art (TILT, Aiger et al): Rectifies patterns that are: a small part of the image, of low
texture Localizes pattern automatically Affine distortion can be removed with as few as 1 repeat
Explicitly model the pattern Segmentation of pattern at pixel-level SIFT variance decreased by using refined pattern to resample
image Multiple motifs can be used to jointly optimize rectification
38/25
3 Apr. PRCV 2014 – J. Pritts, O. Chum, J. Matas: Detection, Rectification, and Segmentation of Co-planar Repeated Patterns
Questions
Thanks for your attention
***Cosegmentations performed by J. Cech, J. Matas, and M. Perdoch. Efficient sequential correspondence selection by cosegmentation. In CVPR, 2008.
39/25