Solving a Special Type of Jigsaw Puzzles: Banknote …szhang16/paper/TMM14.pdf · Imag., vol. 15,...

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 16, NO. 2, FEBRUARY 2014 571

[11] E. D.Montag, “Empirical formula for creating error bars for themethodof paired comparison,” J. Electron. Imag., vol. 15, no. 1, pp. 010 502-1–010 502-3, Mar. 2006.

[12] G. Simons and Y.-C. Yao, “Asymptotics when the number of param-eters tends to infinity in the Bradley-Terry model for paired compar-isons,” Ann. Statist., vol. 27, no. 3, pp. 1041–1060, 1999.

[13] Q. Xu, Q. Huang, T. Jiang, B. Yan,W. Lin, and Y. Yao, “HodgeRank onrandom graphs for subjective video quality assessment,” IEEE Trans.Multimedia, vol. 14, no. 3, pp. 844–857, Jun. 2012.

[14] J. Li, M. Barkowsky, and P. L. Callet, “Analysis and improvementof a paired comparison method in the application of 3DTV subjectiveexperiment,” in Proc. ICIP, Orlando, FL, USA, 2012, pp. 629–632.

[15] M. A. Elliott, “Selecting numerical scales for pairwise comparisons,”Reliab. Eng. Syst. Safety, vol. 95, pp. 750–763, 2010.

[16] R. A. Bradley and M. E. Terry, “Rank analysis of incomplete blockdesigns: I. The method of paired comparisons,” Biometrika, vol. 39,pp. 324–345, 1952.

[17] H. A. David, The Method of Paired Comparisons. London, U.K.:Charles Griffin, 1969.

[18] J. I. Marden, Analyzing and Modeling Rank Data. London, U.K.:Chapman & Hall, 1995.

[19] A. K. Moorthy, K. Seshadrinathan, R. Soundararajan, and A. C. Bovik,“Wireless video quality assessment: A study of subjective scores andobjective algorithms,” IEEE Trans. Circuits Syst. Video Technol., vol.20, no. 4, pp. 587–599, Apr. 2010.

[20] K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack,“Study of subjective and objective quality assessment of video,” IEEETrans. Image Process., vol. 19, no. 6, pp. 1427–1441, Jun. 2010.

[21] M. G. Kendall and B. B. Smith, “On the method of paired compar-isons,” Biometrika, vol. 31, no. 3/4, pp. 324–345, Mar. 1940.

[22] K.-T. Chen, C.-C. Wu, Y.-C. Chang, and C.-L. Lei, “A crowdsource-able QoE evaluation framework formultimedia content,” inProc. ACMMultimedia, Beijing, China, Oct. 2009, pp. 491–500.

[23] S.-J. Kim, C.-B. Chae, and J.-S. Lee, “Quality perception of coding ar-tifacts and packet loss in networked video communications,” in Proc.Workshop Quality of Experience for Multimedia Communications,Anaheim, CA, USA, Dec. 2012, pp. 1357–1361.

[24] N. Ponomarenko, V. Lukin, A. Zelensky, K. Egiazarian, J. Astola, M.Carli, and F. Battisti, “TID2008- a database for evaluation of full-refer-ence visual quality assessment metrics,” Adv. Modern Radioelectron.,vol. 10, pp. 30–45, 2009.

[25] M. Cattelan, “Models for paired comparison data: A review with em-phasis on dependent data,” Statist. Sci., vol. 27, no. 3, pp. 412–433,2012.

Solving a Special Type of Jigsaw Puzzles: BanknoteReconstruction From a Large Number of Fragments

Hongsheng Li, Yuanjie Zheng, Shaoting Zhang, and Jian Cheng

Abstract—In this paper, we propose a method to solve a special type ofjigsaw puzzles, reconstructing banknotes from a large number of fragmentsbased on fragments’ images. Existing jigsaw puzzle assembly algorithmshave difficulty solving this problem effectively. A main limitation of thesemethods is that they do not leverage the following important observations:1) an intact banknote’s image is known and thus can be used as prior in-formation; 2) if two aligned fragments overlap each other, they must notbe from a same banknote. Based on these two important observations, athree-step method is proposed to reconstruct banknotes from their frag-ments. Each fragment is first aligned to its original position on the banknoteby a RANSACmethod. After evaluating every two aligned fragments’ rela-tionships, all fragments are embedded into a lower dimensional space andthen clustered into small groups using a modified agglomerative clusteringmethod. Fragments in a same cluster are likely to be from a same banknote.Experiments on both synthetic and real data demonstrate the effectivenessof our proposed method.

Index Terms—Agglomerative clustering, banknote reconstruction,jigsaw puzzle assembly, spectral embedding.

I. INTRODUCTION

In financial and forensic institutes, there is a need for an automatedalgorithm to reconstruct banknotes from a large number of fragments.In a recent news [3], a woman in China with schizophrenia tore fivehundred Chinese 100-yuan (CNY100) banknotes into small pieces. Ittook 12 bank staffs 6 hours to reconstruct only 1 banknote from a largenumber of small fragments (Fig. 1). Another news states that an ex-pert in the Bureau of Investigation of Taiwan took 7 days to recon-struct two hundred mutilated banknotes [2]. There were similar casesin other countries and regions where banknote reconstruction is needed[1], [4]. An automated method to reconstruct banknotes based on ban-knote fragments’ images would save a lot of manpower and help peoplewith mutilated banknotes recoup their financial losses.This problem can be viewed as a special type of jigsaw puzzles. Al-

though automatically solving traditional jigsaw puzzles has been dis-cussed in a large number of literatures in past decades, a closer look atthe banknote reconstruction problem shows that it has certain specialproperties and challenges, and therefore cannot be effectively solved by

Manuscript received July 05, 2013; revised September 28, 2013; acceptedOctober 01, 2013. Date of publication November 20, 2013; date of current ver-sion January 15, 2014. This work was supported in part by the National ScienceFoundation of China (No. 61301269 and 61201271), in part by the ResearchFund for the Doctoral Program of Higher Education (No. 20100185120021),and in part by the Scientific and Technological Cooperation Program with theAcademy of China and Sichuan Province (No. 2012JZ0001). The associate ed-itor coordinating the review of this manuscript and approving it for publicationwas Dr. Cees Snoek.H. Li and J. Cheng (corresponding author) are with the Department of

Information Engineering, University of Electronic Science and Technology ofChina, Chengdu, Sichuan 611731, China (e-mail: [email protected],[email protected]).Y. Zheng is with the Department of Radiology, University of Pennsylvania,

Philadelphia PA 19014 USA (e-mail: [email protected]).S. Zhang is with the Department of Computer Science, University of North

Carolina at Charlotte, Charlotte, NC 28223 USA (e-mail: [email protected]).Color versions of one or more of the figures in this paper are available online

at http://ieeexplore.ieee.org.Digital Object Identifier 10.1109/TMM.2013.2291968

1520-9210 © 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

572 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 16, NO. 2, FEBRUARY 2014

Fig. 1. According to the news [3], 12 bank staffs spent 6 hours reconstructingonly 1 banknote from the fragments of 500 banknotes.

existing algorithms. In this paper, we primarily focus on the reconstruc-tion of Chinese banknotes, but the proposed method can be generalizedto banknotes from other countries or regions with small modifications.

A. Background

The first computer-aided jigsaw puzzle assembly method [10] wasproposed nearly 50 years ago. Early work primarily uses jigsaw pieces’shape information to determine their neighboring pieces.Wolfson et al.[27] used a well-known human strategy that first identifies and matchesborder pieces by solving a traveling salesman problem. The puzzlepieces in the middle were then matched by a -best method with back-tracking strategy. Goldberg et al. [12] followed the same border-firststrategy but applied a number of new techniques to obtain more ro-bust assembly results. To reduce the search space for pairwise piecematching, Webster et al. [26] detected isthmus points on borders ofpuzzle pieces using medial axis transformation. Those points are usedto describe shape characteristics of puzzle pieces. Kong et al. [14] pro-posed a two-stage method for puzzle piece assembly. In the first stage,candidate matching pairs of pieces are recovered based on local shapeinformation. In the second stage, the overall puzzle is solved by a re-cursive grouping of triples using a best-first with backtracking strategy.Color cues were first incorporated with shape features to better score

local matchings between pairs of puzzle pieces in [15], where a globalsolution is obtained by using the method in [27]. Chung et al. [8] usedboth chromatic information and geometric shape for pairwise puzzlepiece matching. Three different global puzzle assembly algorithmswere proposed and compared. In [29], corner points of each puzzlepiece are extracted and used for recognizing piece types. Pairwiserelationships between pieces are calculated by using boundary shapematching and image merging. In [18], at each characteristic point ofa puzzle piece’s boundary, a set of color and geometrical featuresare extracted for matching puzzle pieces. A greedy algorithm is usedto assemble all pieces together. In [24], the image pattern outsideeach puzzle piece is predicted by inpainting and texture synthesistechniques. Features of each puzzle piece are derived from bothoriginal and predicted image patterns of the piece. In [21], the authorsproposed to use only image features to assemble puzzle pieces. Ageneral puzzle solving method is used to obtain final assembly results.Computational jigsaw assembly has applications in reconstructing

shredded documents or photographs [13], [16], [23]. In [13], theauthors proposed a method to reconstruct shredded documents forforensic purposes. Each fragment’s boundary is approximated bya polygon to lower the computational complexity for matching. A

greedy algorithm is used to assemble fragments one at a time. In [16],a method to automatically assemble shredded pieces from multiplephotos is proposed. A boundary matching score between every twopieces is calculated using both shape and image information. Thepieces are then clustered based on those scores and are assembledvia finding a spanning tree for each cluster. Richter et al. [23] usedthe same data preprocessing techniques as those of [16], but utilizedSupport Vector Machine to find robust matching fragment pairs. Frag-ments were then aligned and assembled via constructing a spanningtree. The above mentioned methods try to solve canonical jigsawpuzzles, where puzzle pieces’ shape information is a strong clue forevaluating pairwise affinities between them. In recent years, muchresearch has focused on a more challenging problem setup, where allpuzzle pieces are squares [11], [22], [28], [5], [7].

B. Our Proposed Method

To the best of our knowledge, there is no previous algorithm specif-ically designed to solve the banknote reconstruction problem. Thisproblem has two properties. Observation 1: an intact banknote’s imageis known and thus can be used as prior information. Observation 2: iftwo aligned pieces overlap each other, they must not be from a samebanknote. Taking advantages of these two properties, we propose athree-step method to effectively reconstruct banknotes from a largenumber of fragments. 1) Based on Observation 1, we propose to usethe -estimator SAmple and Consensus (MSAC) [25], to recover eachfragment’s original orientation and position on the intact banknote. 2)Taking advantages of Observation 2, a spectral technique is used toembed all aligned fragments into a lower dimensional space, whereoverlapping fragments are embedded to positions that are far awayfrom each other. 3) A modified agglomerative clustering method isapplied to all embedded fragments to cluster fragments from samebanknotes into same groups.

II. AUTOMATED BANKNOTE RECONSTRUCTION

A. Preprocessing and Fragment Alignment

Before banknotes can be reconstructed, banknote fragments’ imagesneed to be digitized. This can be done by using either a camera or ascanner. To ease the extraction of fragment images from a digitizedimage, it is better to arrange fragments separately against a single-col-ored background with high contrast. Therefore, we choose to use ascanner with a black background board. The scanning resolution isfixed to 200 dpi. The image of an intact CNY100 banknote is approxi-mately of the size 1210 610.For the frontal-side image of each Chinese banknote, there is a se-

rial number (red rectangles in Fig. 2(a)) and no texture information inthe watermark region (blue polygons in Fig. 2(a)). In contrast, in theback-side images of Chinese banknotes, there are special texture pat-terns designed for anti-faking purposes in the watermark region (bluepolygons in Fig. 2(b)). Those patterns can provide more informationfor the subsequent fragment alignment step. Therefore, back-side im-ages of Chinese banknotes are used in this paper for reconstructionpurposes.Existing methods test every two fragments’ all relative orientations

to calculate their pairwise affinities. However, in our case, Observa-tion 1 states that an intact banknote’s image is known. Each fragment’sorientation and position on the intact banknote could be recovered bymatching the fragment’s image to the intact banknote’s image. To aligneach fragment to its original position, SIFT feature points [17] arefirst extracted from both the fragment image and the intact banknote’simage. For each SIFT point in the fragment image, the Fast Approxi-mate Nearest Neighbors method [19] is used to search its approximate


Fig. 2. (a) The frontal-side images of Chinese banknotes. On every banknote,there is a serial number (as shown by red rectangles) and no image pattern insidethe watermark region (as shown by blue polygons). (b) The back-side images ofChinese banknotes. There are special patterns designed for anti-faking purposesin the watermark regions (as shown by blue polygons). Therefore, using back-side images results in more robust alignment results for small fragments fromthe watermark regions.

Fig. 3. (a) A synthetic fragment. (b) The fragment’s alignment result. Theshaded region illustrates an intact CNY100 banknote’s size.

nearest neighbor from all SIFT feature points [17] in the intact ban-knote’s image. Each SIFT point and its approximate nearest neighborare then stored as a candidate matching pair. One RANSAC variant,the -estimator SAmple and Consensus (MSAC) [25], is then used torecover the rigid transformation between the fragment and the intactbanknote’s image using these candidate matching pairs. One exampleof fragment alignment is shown in Fig. 3.

B. Pairwise Relationships Between Fragments

Existing jigsaw puzzle assembly methods only consider whether apair of fragments adjoin each other or not. However, for the banknotereconstruction problem, overlapping relationships between alignedfragments (as stated by Observation 2) should also be used as an im-portant prior. Therefore, three types of pairwise relationships betweenaligned fragments are considered in our method: 1) if two alignedfragments overlap each other, they must not be from a same banknote;2) if two fragments adjoin each other, they might be from a samebanknote; 3) if two fragments neither adjoin nor overlap each other,we cannot determine whether they are from a same banknote or not.For all aligned fragments, their boundaries are extracted to determine

their pairwise relationships. A thresholding operation is first performedon a fragment image to obtain its foreground mask. Its boundary canthen be obtained by subtracting foreground mask with its morphologi-cally eroded version (see Fig. 4(a)), where the erosion mask’s is chosenas a 3 3 square. A signed distance map is calculated by applying Eu-clidean Distance Transform (DT) to each fragment’s boundary, wheredistance values are positive outside the boundary and are negative in-side the boundary (see Fig. 4(b)).

Fig. 4. (a) The extracted boundary of the aligned fragment in Fig. 3(b). (b) Thesigned distance map of Fig. 3(a), where different colors represent different dis-tance values and the white curve represents the zero level-set of the distancemap.

Given two aligned fragments’ boundary points and signed dis-tance maps, their relationship could be determined in the followingway. Let and represent the two fragments, and betheir boundaries’ signed distance maps, and

denote all boundary points of fragment andfragment , respectively. Please note that , , and 2D indicesof and are all mapped to the intact banknote’s coordinatesystem. For each boundary point of fragment , we obtain itscorresponding distance value in the distance map of fragment as

, which denotes the signed nearest distance between andfragment ’s boundary points.We determine that two fragments overlap each other if

(1)

This condition states that if one fragment’s any boundary point is in-side the other fragment and is pixels away from the boundary, thenthe two fragments overlap each other. We show one example of twooverlapping fragments in Fig. 5(a).Similarly, we also need to determine if two fragments adjoin each

other. Let be the number of fragment ’s boundary points thatare no further than pixels away from the boundary of fragment :

(2)

where is the indication function, i.e., and. is defined similarly as . The smaller

one of and , , is used tocalculate the confidence that the two fragments adjoin eachother,

if ,if .if ,

(3)

where and are lower and upper thresholds to determine the con-fidence that fragments and adjoin each other. The above equa-tion denotes that if the number of matching points on two fragments’boundaries is less than , then the two fragments are determinedas not adjoining each other (i.e., ); if is greater than ,then the two fragments are determined as being very likely to adjoineach other (i.e., ); if is between the two thresholds,the confidence increases from 0 to 1 linearly as increasesfrom to . The confidence function is illustrated in Fig. 6. However,the definition of confidence is not limited to the piecewise linear formmentioned above, other non-decreasing function can also be used. Notethat the adjoining relationship is overruled by the previous overlapping


Fig. 5. Two example fragments (marked by colored curves) and their pairwise relationships with the fragment in Fig. 3. (Row 1) Aligned fragments and theiroriginal positions on the intact banknote. (Row 2) Illustration of using boundary points and signed distance maps to determine pairwise relationships betweenaligned fragments. (a) The second fragment (marked by red and green curves) overlaps the fragment in Fig. 3 (marked by blue curves). The red curve representsthe second fragment’s boundary points that are inside the boundary of the fragment in Fig. 3. (b) The second fragment (marked by red and green curves) adjoinsthe fragment in Fig. 3 (marked by blue curves). The red curve represents the boundary points adjoining two fragments. (c) Two fragments (marked by blue curves)that neither overlap nor adjoin each other. (a) Overlapping Relation. (b) Adjoining Relation. (c) Unknown Relation.

Fig. 6. The confidence that fragments and adjoin each other.

relationship. One example of two adjoining fragments are shown inFig. 5(b).If two fragments meet neither of the above conditions, (1) and (3),

then we cannot determine whether they are from a same banknote ornot. In practice, many pairwise relationships fall into this category andcan therefore be quickly determined based on two fragments’ align-ment results. One example of two adjoining fragments that neitheroverlap nor adjoin each other is shown in Fig. 5(c).When banknotes are shredded by hand, adjoining fragments’ bound-

aries might overlap each other. In such cases, the parameters andshould be adjusted accordingly. Both and should be increasedto accommodate the boundaries of “none-fine-cut” fragments. By in-creasing in (1), it is less sensitive to determine if two fragmentsoverlap each other. By increasing in (2), it is less sensitive to de-termine if a boundary point of a fragment does not match the boundaryof another fragment. Therefore, they can better tolerate the boundariesof none-fine-cut fragments.

C. Spectral Embedding With Attraction and Repulsion

Our goal is to cluster fragments from same banknotes into samegroups with all available pairwise-relationship information. To achievethis goal, all fragments are first embedded into a new space such thatfragments from same banknotes are close to each other while fragments

from different banknotes are far away from each other. Clustering insuch a new space is therefore easier to perform and can obtain moreaccurate clustering results. We propose to use the spectral techniquewith attraction and repulsion proposed in [30] to embed all fragmentsin a lower dimensional space.Let be the number of all scanned fragments. Their pairwise rela-

tionships are modeled by an undirected weighted graph ,in which vertex models the th fragment and the edge

between vertex and is associated with two weights,a repulsion weight and an attraction weight . In our pro-posed method, we set if fragments and overlap eachother, and otherwise. Similarly, we setif fragments and adjoin each other, and otherwise.We can define two matrices, and , where

and are stored at the th row and th column of and, respectively. Let and represent diagonal degree matrices,

where and .Let be all fragments’ embedding coordinates in a lower-dimensional space, i.e., . The th row of , , represents

fragment ’s coordinates in the embedding space. Ideally, all frag-ments’ positions in the embedding space, , should be faraway from each other if repulsions between them are large (i.e., max-imizing ), and should be close to each otherif attractions between them are large (i.e., minimizing

). Let

(4)

(5)

The above two goals can be fulfilled simultaneously by optimizing thefollowing objective function,

(6)

where is a scaled partition matrix. Theabove constraint removes an arbitrary scaling factor in the embeddingresults. Using the method of Lagrange multipliers, the above quadratic


formulation leads to the standard generalized eigenvalue problem:the maximizer of the above optimization model, , isgiven by the largest generalized eigenvectors of .The optimal embedding for all fragments can then be calculated as

.D. Modified Agglomerative Clustering for Banknote Reconstruction

Traditional spectral clustering methods utilize -means after the em-bedding is performed. However, the number of banknotes (i.e., thevalue of ) is usually not known in advance. Moreover, we would likethe clustering algorithm to constrain the total area of fragments in onecluster not exceeding that of an intact banknote. We propose to modifythe graph-based Agglomerative Clustering method [9] to group frag-ments from same banknotes into same clusters. Algorithm 1 summa-rizes our proposed clustering method, where denote theareas of all fragments, and denotes the area of an intact banknote.

Algorithm 1: Modified Agglomerative Clustering for BanknoteReconstruction

Input: Fragments’ embedding coordinates ;Fragments’ areas ; Repulsion matrix ; Attractionmatrix ;

Output: Clusters containing fragments from same banknotes

begin

Step 1: Construct a graph ;

Step 2: repeat

Step 2.1: Find two best vertices to be merged;

Step 2.2: Merge ;

Step 2.3: Update graph ;

until no mergeable vertices in ;

return all vertices in ;

end

1) Construct a New Graph: a fully-connected and weighted graphis constructed, where each vertex represents

a cluster containing only one fragment, and there is an edgebetween every pair of vertices and . Each vertex is associatedwith coordinates representing the center of the cluster, a param-eter representing the total area of fragments in the cluster, and aparameter representing the number of fragments in the cluster.Each edge is associated with a repulsion weight ,and a distance weight as the Euclidean distance be-tween the centers of the two vertices if .2) Iteratively Merge Vertices: at each iteration, the best pair of ver-

tices is chosen for merging. We search for the edgesuch that it has the smallest Euclidean distance while its repulsionweight and the total area of fragments in the two vertices

. The two vertices and are merged in graph tocreate a new one, . We calculate ’s cluster center, the number of fragments and the total area of fragments

in vertex as

(7)

(8)

(9)

For each vertex with an edge connecting to or , a new edgeis created between the vertices and , whose repulsion

weight is set as

(10)

If , its distance weight is calculated as

(11)

3) Terminate Iterations: if there are no more mergeable vertices in, the algorithm terminates iterations and returns all vertices (clusters)

in . Since all fragments are aligned to their original locations, theycan be easily assembled based on our proposed method’s clusteringresults.

E. Connections to the Shredded Photo Assembly Method [16]

Our method is related to the shredded photo assembly method in[16], both of which use spectral techniques to embed fragments intoa lower dimensional space for easier clustering. However, our methodsuccessfully utilizes all available information in the banknote recon-struction problem to constrain the assembly results.In the embedding step, compared with the photo reconstruction

method in [16], our proposed method utilizes repulsion relationshipsbetween overlapping fragments, which is crucial for embedding over-lapping fragments to positions that are far away from each other. Inthe clustering step, we propose a modified Agglomerative Clusteringmethod which could better constrain the grouping results. The pro-posed clustering method only clusters fragments together if there is norepulsion between them, and does not allow to form a cluster whosefragments’ total area exceeds that of an intact banknote. However, themethod in [16] cannot utilize such prior knowledge to constrain theclustering results.

III. EXPERIMENTS

A. Parameter Settings, the Compared Method, and EvaluationCriterion

For our fragment alignment method in Section II-A, we set MSAC’sinlier threshold to 3 pixels and let the algorithm run for 10000 iter-ations. Using a single thread on an Intel i7 860 2.80 GHz CPU, ourC++ implementation takes less than 1s to align each fragment. To mea-sure the performance of our proposed fragment alignment method, wedetermined whether a fragment was successfully aligned if its recov-ered rotation and translation were within 0.5 and 2.5 pixels of theground-truth rotation and translation, respectively. For fragments withno ground-truth transformation parameters, the per-pixel-differencesbetween the aligned fragment and the intact banknote’s image were cal-culated. If 50% of the aligned fragment’s pixel-differences were lessthan a given threshold, we considered the fragment was successfullyaligned. To determine pairwise relationships between fragments, therelated parameters are fixed to , , andwhen the scanning resolution is set to 200 dpi.We compared our spectral embedding and modified agglomerative

clustering method with the spectral clustering method [20] used in theshredded photo assembly [16], which tries to maintain only affinitiesbetween training samples specified by the attraction matrix . The-means++ algorithm [6] was used to choose good initial centers forthe -means clustering step in [20]. For each case, 10 different initial-izations were generated by -means++ and the clustering result withthe smallest average distance to cluster centers was chosen as the finalresult.


For the spectral clustering method [20], it requires the number ofbanknotes (i.e., the for the -means clustering step) to be specified. Incomparison, our proposed method only needs the dimension numberof the embedding space. Although theoretically, should equal , ourexperiments show that approximate estimates of result in satisfactoryclustering performance.To measure the accuracy of clustering results, the results of each

pair of fragments are evaluated: a true positive (TP) decision assignstwo fragments from a same banknote to a same cluster; a true nega-tive (TN) decision assigns two fragments from different banknotes todifferent clusters; a false positive (FP) decision assigns two fragmentsfrom different banknotes to a same cluster; a false negative (FN) deci-sion assigns two fragments from a same banknote to different clusters.Let , , , and denote the numbers of TPs, TNs,FPs, and FNs, respectively. The Rand Index ( ) and the score arecalculated as

(12)

(13)

to evaluate the clustering results by the two methods.

B. A Toy Example

We first illustrate the differences between the two methods using atoy example shown in Fig. 7, where fragments , and , were fromtwo identical images. Pairwise attraction and repulsion relationshipsbetween fragments were set to either 0 or 1 for illustrative purposes.The attraction and repulsion matrices were specified as follows:

(14)

In this example we assumed that the attraction between fragmentsand was mistakenly calculated as 1 because some of their boundariescan be exactly matched, i.e., . The clusteringresults obtained by spectral clustering [20] and by our proposedmethod are shown in Fig. 7(c) and (d), which illustrate that whenthere are incorrect attraction relationships, the spectral clustering[20] might embed fragments from different banknotes to positionsthat cannot be easily separated. In contrast, our proposed methodcan generate easy-to-cluster embedding results with the additionalrepulsion information.

C. Synthetic Data

We performed experiments using a large number of synthetic ban-knote fragments. Given a CNY100 banknote’s template image, we ran-domly segmented it into 10 regions and randomly rotated each regionto create a synthetic banknote fragment. Such operation was repeatedfor 200 times and resulted in 2000 fragments belonging to 200 dif-ferent clusters. One such synthetic fragment is illustrated in Fig. 3(a).All 2000 synthetic fragments were successfully aligned to their orig-inal positions on the banknote. The aligned fragments of the first 5, 10,50, 100, and all 200 banknotes were used to create the attraction matrixand repulsion matrix . The two matrices are used as the inputs for

the two methods.For the spectral clustering method [20], we set to the exact number

of banknotes. This is because an incorrect value would never leadto a correct clustering result. For our proposed method, is set to

, , and respectively to test the sensitivity of incorrectestimates of . The Rand Index ( ) and score of the clustering

Fig. 7. A toy example illustrating the differences between the spectral clus-tering method in [20] and our proposed method. (a-b) Fragments , , and ,forms two identical images. Their pairwise attractions and repulsions are spec-ified in (14). (c) The embedding results using both pairwise attractions andrepulsions by our proposed method. Fragments , and , can be easilyseparated after embedding. (d) The embedding results using only pairwise at-tractions by the spectral clustering method [20]. Note that fragments andare embedded to the same position.

TABLE IRAND INDEX ( ) AND SCORE OF CLUSTERING RESULTS

FOR SYNTHETIC DATA BY OUR PROPOSED METHODAND BY THE SPECTRAL CLUSTERING METHOD [20]

results by the two methods are recorded in Table I. When the numberof banknotes is small (less than 10), both methods are able to generatesatisfactory clustering results; as the number of banknotes increasesfrom 50 to 200, our proposed method outperforms the spectral clus-tering [20] by a large margin.

D. Real Data

We then performed experiments using real fragments from prac-ticing banknotes (because either printing banknote images or shred-ding real banknotes is illegal in China). Those practicing banknotesare used for bank staffs to train their counting skills. They have com-plex image patterns and are therefore suitable for testing our proposedmethod. Two types of practicing banknotes were used in our experi-ments (see Fig. 8(a) and (b)). For the first type (Fig. 8(a)), we scissored100 such banknotes, each of which was shredded into 7–15 pieces (see


Fig. 8. (a-b) Two types of practicing banknotes used in our experiments asdescribed in Section III-D . (c) The fragments of 100 practicing banknotes of(a) used in our real-data experiment. (d) Six reconstructed banknotes guided byour proposed method’s clustering results.

Fig. 9. (First row) One example misalignment case due to not enough texturedpatterns. (Second row) One example misalignment case due to too small frag-ment size. (a) Before alignment. (b) After alignment.

Fig. 8(c)); for the second type (Fig. 8(b)), we shredded 50 such ban-knotes by hand, each of which was shredded into 5–8 pieces. All frag-ments were marked with identification numbers in the back and com-pletely shuffled before scanning.

TABLE IIRAND INDEX ( ) AND SCORE OF CLUSTERING RESULTS

FOR REAL DATA BY OUR PROPOSED METHODAND BY THE SPECTRAL CLUSTERING METHOD [20]

Using our fragment alignment method in Section II-A, 1 out of 1013fragments of the red banknotes and 5 out of 335 fragments of the graybanknotes were unsuccessfully aligned. The unsuccessfully alignedones eventually result in one-fragment clusters. There are two mainfactors that cause failure cases by our proposed alignment method:1) if the fragment does not have enough textured patterns, then theremight not be enough feature points extracted from the fragment’simage to guide the alignment. One such failure case is illustrated inthe first row of Fig. 9. 2) If the fragment is of too small size, featurepoints’ feature vectors might describe too much background and thusbecome unreliable for point matching. We illustrate one such failurecase in the second row of Fig. 9.Similar to the previous experiment, we set , ,

and respectively to test the robustness of our proposed methodagainst incorrect estimates of . The clustering results by both methodswere evaluated by the Rand Index ( ) and the score (see Table II),which again show that our proposed method outperforms the methodin [20] for the real banknote reconstruction task. With the guidance ofour method’s clustering results, all shredded banknotes were manuallyassembled by four persons in 3 hours. Six example reconstructedbanknotes are shown in Fig. 8(d).

IV. DISCUSSIONS AND CONCLUSIONS

Although the Chinese banknotes have textured patterns in their wa-termark regions, banknotes from other countries may have texture-lessregions. Fragments from these regions might not be aligned correctlyusing our proposed feature-based alignment method in Section II-A.To solve this problem, textured fragments should be aligned first. Suchtexture-less fragments’ relationships with other fragments can then beestimated using shape cues as those in [16], [23].In our experiments, our proposed method significantly outperforms

the spectral clustering method [20] when the number of banknotes isgreater than 50. Two factors contribute to the improvements: 1) thespectral clustering method [20] does not take overlapping relationshipsbetween fragments into account and thus might cluster overlappingfragments into same clusters. In contrast, our method embeds over-lapping fragments to positions that are far away from each other andexplicitly forbids clustering such fragments into same groups. 2) The-means clustering, as the last step of the spectral clustering method[20], can only recover a local optimum and is not robust when islarge. In comparison, our proposed agglomerative clustering methoduses prior knowledge about both fragments’ pairwise relationships andsize information to constrain cluster results.In summary, we presented a method for reconstructing banknotes

from a large number of fragments using spectral embedding and ag-glomerative clustering techniques. Two properties of this problemwereobserved and used as prior information in our proposed method. Exper-iments and comparison using both synthetic and real data have demon-strated the effectiveness of our proposed method for the banknote re-construction task. In the future, the proposed method will be furthervalidated using banknotes from other counties or regions.


REFERENCES[1] [Online]. Available: http://www.bot.or.th/english/banknotes/bank-

note%20service/pages/claims.aspx[2] 2010 [Online]. Available: http://www.fjsen.com/b/2010–12/10/con-

tent_3916664.htm[3] 2012 [Online]. Available: http://www.nownews.com/2012/05/05/

91–2810990.htm[4] 2013 [Online]. Available: http://blogs.reuters.com/photogra-

phers-blog/2013/08/01/euro-in-pieces/[5] N. Alajlan, “Solving square jigsaw puzzles using dynamic program-

ming and the Hungarian procedure,” Amer. J. Appl. Sci., vol. 6, no. 11,pp. 1941–1947, 2009.

[6] D. Arthur and S. Vassilvitskii, “ -means++: the advantages ofcareful seeding,” in Proc. ACM-SIAM Symp. Discrete Algorithms, pp.1027–1035.

[7] T. S. Cho, S. Avidan, andW. T. Freeman, “A probabilistic image jigsawpuzzle solver,” inProc. IEEEConf. Comput. Vision and Pattern Recog-nition, 2010.

[8] M. G. Chung, M. M. Fleck, and D. A. Forsyth, “Jigsaw puzzle solverusing shape and color,” in Proc. Int. Conf. Signal Process., 1998, pp.877–880.

[9] P. Franti, O. Virmajoki, and V. Hautamaki, “Fast agglomerative clus-tering using a -nearest neighbor graph,” IEEE Trans. Pattern Anal.Mach. Intell., vol. 28, no. 11, pp. 1875–1881, 2006.

[10] H. Freeman and L. Gardner, “Apictorial jigsaw puzzles: The computersolution of a problem in pattern recognition,” IEEE. Trans. Electron.Comput., vol. 2, pp. 118–127, 1964.

[11] A. C. Gallagher, “Jigsaw puzzles with pieces of unknown orientation,”in Proc. IEEE Conf. Comput. Vision and Pattern Recognition, 2012.

[12] D. Goldberg, C. Malon, and M. Bern, “A global approach to automaticsolution of jigsaw puzzles,” in Proc. Symp. Comput. Geometry, 2002,pp. 82–87.

[13] E. Justino, L. Oliveria, and C. Freitas, “Reconstructing shredded docu-ments through feature matching,” Forensic Sci. Int., vol. 160, no. 2–3,pp. 140–147, 2006.

[14] W. Kong and B. B. Kimia, “On solving 2D and 3D puzzles using curvematching,” in Proc. IEEE Conf. Comput. Vision and Pattern Recogni-tion, 2001.

[15] D. A. Kosiba, P. M. Devaux, S. Balasubramanian, T. L. Gandhi, andK. Kasturi, “An automatic jigsaw puzzle solver,” in Proc. Int. Conf.Pattern Recognition, 1994, pp. 616–618.

[16] H. Liu, S. Cao, and S. Yan, “Automated assembly of shredded piecesfrom multiple photos,” IEEE Trans. Multimedia, vol. 13, no. 5, pp.1154–1162, 2011.

[17] D. G. Lowe, “Distinctive image features from scale-invariant key-points,” Int. J. Comput. Vision, vol. 60, no. 2, pp. 91–110, 2004.

[18] M. Makridis and N. Papamarkos, “New technique for solving a jigsawpuzzle,” in Proc. Int. Conf. Image Process., 2006, pp. 2001–2004.

[19] M. Muja and D. G. Lowe, “Fast approximate nearest neighbors withautomatic algorithm configuration,” in Proc. Int. Conf. Comput. VisionTheory and Applicat., 2009, pp. 331–340.

[20] A. Ng, M. Jordan, and Y. Weiss, “On spectral clustering: analysis andan algorithm,” in Proc. Int. Conf. Advances in Neural Inform. Process.Syst., 2002, pp. 849–856.

[21] T. R. Nielsen, P. Drewsen, and K. Hansen, “Solving jigsaw puzzlesusing image features,” Pattern Recognit. Lett., vol. 29, no. 14, pp.1924–1933, 2008.

[22] D. Pomeranz, M. Shemesh, and O. Ben-Shahar, “A fully automatedgreedy square jigsaw puzzle solver,” in Proc. IEEE Conf. Comput. Vi-sion and Pattern Recognition, 2011.

[23] F. Richter, C. X. Ries, N. Cebron, and R. Lienhart, “Learning to re-assemble shredded documents,” IEEE Trans. Multimedia, vol. 15, no.3, pp. 582–593, 2013.

[24] M. Sairoglu and A. Ercil, “A texture based matching approach for au-tomated assembly of puzzles,” in Proc. Int. Conf. Pattern Recognition,2006, pp. 1036–1041.

[25] P. H. S. Torr and A. Zisserman, “MLESAC: A new robust estimatorwith application to estimating image geometry,” Comput. Vision ImageUnderstand., vol. 78, no. 1, pp. 138–156, 2000.

[26] R. Webster, P. LaFollette, and R. Stafford, “Isthmus critical points forsolving jigsaw puzzles in computer vision,” IEEE Trans. Syst., Man,Cybern., vol. 21, pp. 1271–1278, 1991.

[27] H. Wolfson, E. Schonberg, A. Kalvin, and Y. Lamdan, “Solving jigsawpuzzles by computer,” Ann. Oper. Res., vol. 12, pp. 51–64, 1988.

[28] X. Yang, N. Adluru, and L. Latecki, “Particle filter with state permuta-tions for solving image jigsaw puzzles,” in Proc. IEEE Conf. Comput.Vision and Pattern Recognition, 2011.

[29] F.-H. Yao and G.-F. Shao, “A shape and image merging techniqueto solve jigsaw puzzles,” Pattern Recognit. Lett., vol. 24, no. 12, pp.1819–1835, 2002.

[30] S. X. Yu and J. Shi, “Understanding popout through repulsion,” inProc.IEEE Conf. Comput. Vision and Pattern Recognition, 2001.

Solving a Special Type of Jigsaw Puzzles: Banknote …szhang16/paper/TMM14.pdf · Imag., vol. 15,...

Documents

Transcript of Solving a Special Type of Jigsaw Puzzles: Banknote …szhang16/paper/TMM14.pdf · Imag., vol. 15,...