Post on 22-Jun-2020
A Fully-Automated Solver for Multiple SquareJigsaw Puzzles Using Hierarchical Clustering
October 27, 2016
Zayd Hammoudehhammoudeh@gmail.com
Department of Computer ScienceSan José State University
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
1Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
IntroductionThesis Goals
Primary Goal: Develop a solver that can assemble multiplejigsaw puzzles simultaneously, with performance that exceedsthe state of the art.
Additional Goals:
I Define the first metrics that quantify the quality of outputsfrom a multi-puzzle solver
I Design visualizations for viewing the errors (if any) in asolver output
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
1Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
IntroductionThesis Goals
Primary Goal: Develop a solver that can assemble multiplejigsaw puzzles simultaneously, with performance that exceedsthe state of the art.
Additional Goals:
I Define the first metrics that quantify the quality of outputsfrom a multi-puzzle solver
I Design visualizations for viewing the errors (if any) in asolver output
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
2IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
IntroductionJigsaw Puzzles
I First jigsaw puzzle introduced in the 1760s. Modern jigsawpuzzles introduced in the 1930s.
I First computational jigsaw puzzle solver introduced in1964
I Solving a jigsaw puzzle is NP-complete [1, 2]
I Example Applications: DNA fragment reassembly,shredded document reconstruction, speech descrambling,and image editing
I In most cases, the original, “ground-truth” image isunknown.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
2IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
IntroductionJigsaw Puzzles
I First jigsaw puzzle introduced in the 1760s. Modern jigsawpuzzles introduced in the 1930s.
I First computational jigsaw puzzle solver introduced in1964
I Solving a jigsaw puzzle is NP-complete [1, 2]
I Example Applications: DNA fragment reassembly,shredded document reconstruction, speech descrambling,and image editing
I In most cases, the original, “ground-truth” image isunknown.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
2IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
IntroductionJigsaw Puzzles
I First jigsaw puzzle introduced in the 1760s. Modern jigsawpuzzles introduced in the 1930s.
I First computational jigsaw puzzle solver introduced in1964
I Solving a jigsaw puzzle is NP-complete [1, 2]
I Example Applications: DNA fragment reassembly,shredded document reconstruction, speech descrambling,and image editing
I In most cases, the original, “ground-truth” image isunknown.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
3IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzles
Jig Swap Puzzles – Variant of the traditional jigsaw puzzleI All pieces are equal-sized squaresI Substantially more difficult
Ground-Truth Image Randomized Jig Swap Puzzle
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
3IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzles
Jig Swap Puzzles – Variant of the traditional jigsaw puzzleI All pieces are equal-sized squaresI Substantially more difficult
Ground-Truth Image
Randomized Jig Swap Puzzle
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
3IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzles
Jig Swap Puzzles – Variant of the traditional jigsaw puzzleI All pieces are equal-sized squaresI Substantially more difficult
Ground-Truth Image Randomized Jig Swap Puzzle
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
4IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzle Types
There are three primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.
I Type 1: Puzzle dimensions and piece rotation are known.Only piece location is unknown.
I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.
I Mixed-Bag: Pieces come from multiple puzzles.
Mixed-Bag puzzles are the focus of this thesis.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
4IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzle Types
There are three primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.
I Type 1: Puzzle dimensions and piece rotation are known.Only piece location is unknown.
I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.
I Mixed-Bag: Pieces come from multiple puzzles.
Mixed-Bag puzzles are the focus of this thesis.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
4IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzle Types
There are three primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.
I Type 1: Puzzle dimensions and piece rotation are known.Only piece location is unknown.
I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.
I Mixed-Bag: Pieces come from multiple puzzles.
Mixed-Bag puzzles are the focus of this thesis.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
4IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzle Types
There are three primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.
I Type 1: Puzzle dimensions and piece rotation are known.Only piece location is unknown.
I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.
I Mixed-Bag: Pieces come from multiple puzzles.
Mixed-Bag puzzles are the focus of this thesis.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
4IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzle Types
There are three primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.
I Type 1: Puzzle dimensions and piece rotation are known.Only piece location is unknown.
I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.
I Mixed-Bag: Pieces come from multiple puzzles.
Mixed-Bag puzzles are the focus of this thesis.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
Introduction5Previous Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Previous WorkPaikin & Tal
Paikin & Tal [4] – Current State of the ArtI Greedy, kernel growing solver
I Supports Type 1, Type 2, and Mixed-Bag puzzles
I Immune to missing pieces
Limitations:I Poor Seed Selection: All decisions are made at runtime
using as few as 13 pieces
I Externally Supplied Information: The solver must betold the number of input puzzles
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
Introduction5Previous Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Previous WorkPaikin & Tal
Paikin & Tal [4] – Current State of the ArtI Greedy, kernel growing solver
I Supports Type 1, Type 2, and Mixed-Bag puzzles
I Immune to missing pieces
Limitations:I Poor Seed Selection: All decisions are made at runtime
using as few as 13 pieces
I Externally Supplied Information: The solver must betold the number of input puzzles
The Mixed-Bag Solver
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
6Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Mixed-Bag SolverBasic Structure
...
Mixed Bag
Final
Assembly
Hierarchical
Segment
Clustering
Segmentation Stitching
Final
Seed Piece
Selection
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
7Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Mixed-Bag SolverFoundations
Paikin & Tal’s AlgorithmI Begin each puzzle with a single piece
I Place all pieces around the expanding kernel
Alternate Jigsaw Puzzle Solving StrategyI Correctly assemble small puzzle regions (i.e., segments)
I Iteratively merge smaller regions to form large ones
I Advantages of this Approach:
I Reduces the size of the problem
I Provides structure to the unordered set of puzzle pieces.
The alternate strategy is the basis of the Mixed-Bag Solver
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
7Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Mixed-Bag SolverFoundations
Paikin & Tal’s AlgorithmI Begin each puzzle with a single piece
I Place all pieces around the expanding kernel
Alternate Jigsaw Puzzle Solving StrategyI Correctly assemble small puzzle regions (i.e., segments)
I Iteratively merge smaller regions to form large ones
I Advantages of this Approach:
I Reduces the size of the problem
I Provides structure to the unordered set of puzzle pieces.
The alternate strategy is the basis of the Mixed-Bag Solver
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
7Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Mixed-Bag SolverFoundations
Paikin & Tal’s AlgorithmI Begin each puzzle with a single piece
I Place all pieces around the expanding kernel
Alternate Jigsaw Puzzle Solving StrategyI Correctly assemble small puzzle regions (i.e., segments)
I Iteratively merge smaller regions to form large ones
I Advantages of this Approach:
I Reduces the size of the problem
I Provides structure to the unordered set of puzzle pieces.
The alternate strategy is the basis of the Mixed-Bag Solver
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
7Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Mixed-Bag SolverFoundations
Paikin & Tal’s AlgorithmI Begin each puzzle with a single piece
I Place all pieces around the expanding kernel
Alternate Jigsaw Puzzle Solving StrategyI Correctly assemble small puzzle regions (i.e., segments)
I Iteratively merge smaller regions to form large ones
I Advantages of this Approach:
I Reduces the size of the problem
I Provides structure to the unordered set of puzzle pieces.
The alternate strategy is the basis of the Mixed-Bag Solver
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
8Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Mixed-Bag SolverOverview
I The Mixed-Bag Solver is fully-automated. It makes noassumptions concerning piece orientation, puzzledimensions, or number of puzzles.
I Input: A bag of puzzle pieces
I Output: One or more disjoint, solved puzzles.
I The Mixed-Bag Solver consists of five distinct stages:
...
Mixed Bag
Final
Assembly
Hierarchical
Segment
Clustering
Segmentation Stitching
Final
Seed Piece
Selection
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag Solver9Assembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
AssemblerMixed-Bag Solver Component
I Role: Place the individual pieces in the solved puzzle.
I Mixed-Bag Solver is independent of the assembler used,giving the solver significant upgradability and flexibility.
I Assembler Used in this Thesis: Paikin & Tal
I Current state of the art
I Allows for more direct comparison of performance
I Natively supports Mixed-Bag puzzles
I Implementation: Assembler re-implemented as part ofthis thesis based off the description in [4]
I Written in Python and fully open source [5]
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag Solver9Assembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
AssemblerMixed-Bag Solver Component
I Role: Place the individual pieces in the solved puzzle.
I Mixed-Bag Solver is independent of the assembler used,giving the solver significant upgradability and flexibility.
I Assembler Used in this Thesis: Paikin & Tal
I Current state of the art
I Allows for more direct comparison of performance
I Natively supports Mixed-Bag puzzles
I Implementation: Assembler re-implemented as part ofthis thesis based off the description in [4]
I Written in Python and fully open source [5]
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
10Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
SegmentationMixed-Bag Solver Stage #1
I Segment: Partial puzzle assembly where this is a highdegree of confidence pieces are placed correctly.
I Role of Segmentation: Provide structure to the set ofpuzzle pieces by partitioning them into disjoint segments
I Input: A bag of puzzle pieces
I Output: Set of saved segments
I Relationship between Puzzle Pieces and Segments:
I Pieces from a single ground-truth input may be separatedinto multiple segments
I A piece can be assigned to at most one segment
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
11Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
SegmentationAlgorithm Overview
I Iterative process consisting of one or more rounds
I In each round, all pieces not yet assigned to a segmentare assembled as if all are from the same input image
I Segments of sufficient size are saved to be used in futureMixed-Bag Solver stages
I Pieces in a saved segment are not placed in future rounds.
I Segmentation terminates if all pieces are assigned to asaved segment or when no segment is larger than theminimum allowed size
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
12Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
SegmentationComposition of a Segment
I Starting a Segment: Each segment is created iterativelystarting with a single seed piece
I Definition of Best Buddies: Any pair of pieces that aremore similar to each other than they are to any other piece.
I Growing the Segment: Add to the segment any piecethat is a neighbor and best buddy of a segment member
I Trimming the SegmentI Articulation Point: Any piece whose removal disconnects
other pieces from the segment seed.I All articulation pieces pieces are removed from the segment.
I After the removal of the articulation points, any pieces nolonger connected to the seed are removed.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
13Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
SegmentationExample – Input Images
Image (a) – 805 Pieces [6]
Image (b) – 540 Pieces [7]
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
14Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
SegmentationExample – First Segmentation Round Output Image
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
15Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
SegmentationExample – Segmented Output Image
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
16Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
StitchingMixed-Bag Solver Stage #2
I Role of Stitching: Quantify the extent that any pair ofsegments is related.
I Input: All puzzle pieces and the set of saved segments
I Output: Segment overlap matrix
I Theoretical Foundation: If two segments are from thesame ground-truth image, they would eventually overlap ifone segment were to expand.
I Segments should be allowed, but not forced, to expand in alldirections.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
16Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
StitchingMixed-Bag Solver Stage #2
I Role of Stitching: Quantify the extent that any pair ofsegments is related.
I Input: All puzzle pieces and the set of saved segments
I Output: Segment overlap matrix
I Theoretical Foundation: If two segments are from thesame ground-truth image, they would eventually overlap ifone segment were to expand.
I Segments should be allowed, but not forced, to expand in alldirections.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
17Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
StitchingStitching Piece Location
I Mini-assembly (MA): Same as a standard assemblyexcept only a fixed number of pieces are placed .
I Stitching Piece (ζx ): A piece near the boundary of asegment that is used as the seed of a mini-assembly
I Segment Overlap: Maximum overlap between anymini-assembly for segment, Φi and another segment Φj .
OverlapΦi ,Φj = arg maxζx∈Φi
|MAζx
⋂Φj |
min(|MAζx |, |Φj |)(1)
I Asymmetry: In most cases:
OverlapΦi ,Φj 6= OverlapΦj ,Φi (2)
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
18Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
StitchingExample – Input Image
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
19Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
StitchingExample – Two Segment Images
Segment #1 Segment #2
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
20Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
StitchingExample – Stitching Piece Locations
Segment #1 Segment #2
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
21Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
StitchingExample – Stitching Piece Locations
Stitching Result from Segment #1
Segment Overlap:
OverlapΦ1,Φ2 = 0.83 (3)
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
22Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Hierarchical Segment ClusteringMixed-Bag Solver Stage #3
I A single ground-truth image may be comprised of multiplesegments.
I Role of Hierarchical Clustering: Merge all segmentsfrom the same input image into a single segment cluster.
I Input: All saved segments and the segment overlap matrix
I Output: A set of segment clusters
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
23Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Hierarchical Segment ClusteringCalculating the Initial Similarity Matrix
I Segment Overlap Matrix: A hollow matrix quantifying therelationship between each pair of segments.
I Hierarchical Clustering Similarity Matrix: A diagonalmatrix quantifying the similarity between segment pairs.
I Quantifying Similarity: Given n segments, the similaritybetween segments Φi and Φj is:
ωi,j =OverlapΦi ,Φj + OverlapΦj ,Φi
2(4)
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
24Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Hierarchical Segment ClusteringMerging Clusters
I After two clusters are combined, the similarity between themerged cluster and all other clusters must be recalculated.
I Single Link Clustering: The similarity between any twoclusters is equal to the maximum similarity between anytwo members in the clusters [8]
I The Mixed-Bag Solver must use single link clustering astwo clusters may only have two member segments that areadjacent.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
25Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Hierarchical Segment ClusteringExample – Single Linking
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
26Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Hierarchical Segment ClusteringExample – Single Linking
Segment 1 Segment 2 Segment 3 Segment 4
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
27Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Hierarchical Segment ClusteringExample – Single Linking
Segment Cluster 1 Segment 3 Segment 4
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
28Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Hierarchical Segment ClusteringExample – Single Linking
Segment Cluster 1 Segment Cluster 2
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
29Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Hierarchical Segment ClusteringTerminating Clustering
I The solver continues merging segment clusters until oneof two criteria is satisfied:
I Only a single segment cluster remains
I Maximum similarity between any segment clusters is belowa predefined threshold
I All remaining segment clusters are passed to the nextsolver stage.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
30Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Final Seed Piece SelectionMixed-Bag Solver Stage #4
Mixed-Bag Solver
I Role of Final Seed Selection: Determine the pieces thatwill be used as the seed for the final output puzzles.
I Input: Set of segment clusters
I Output: Final seed pieces
I A single seed piece is selected from each segment cluster
Paikin & Tal
I All puzzle seeds are selected greedily at run time, whichoften leads to poor decisions.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
30Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Final Seed Piece SelectionMixed-Bag Solver Stage #4
Mixed-Bag Solver
I Role of Final Seed Selection: Determine the pieces thatwill be used as the seed for the final output puzzles.
I Input: Set of segment clusters
I Output: Final seed pieces
I A single seed piece is selected from each segment cluster
Paikin & Tal
I All puzzle seeds are selected greedily at run time, whichoften leads to poor decisions.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
31Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Final Assembly StageMixed-Bag Solver Stage #5
I Role of Final Assembly: Generate the solved puzzlesthat are output by the Mixed-Bag Solver.
I Input: Set of puzzle pieces with the seeds marked
I Output: Final solved puzzles
I All pieces are placed around the seeds selected in theprevious stage.
I Assembly proceeds in this stage normally without anycustom modifications.
Quantifying Solver Quality
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
32Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Quantifying Solver Quality
I Jigsaw puzzle solvers are not able to always correctlyreconstruct the input puzzle(s)
I Metrics compare the quality of solver outputs
I Two Most Common Quality Metrics:
I Direct Accuracy
I Neighbor Accuracy
I Disadvantages of Current Metrics: Neither account for:
I Pieces misplaced in different puzzles
I Extra pieces from other puzzles
I Goal: Define new quality metrics for Mixed-Bag puzzles
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
32Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Quantifying Solver Quality
I Jigsaw puzzle solvers are not able to always correctlyreconstruct the input puzzle(s)
I Metrics compare the quality of solver outputs
I Two Most Common Quality Metrics:
I Direct Accuracy
I Neighbor Accuracy
I Disadvantages of Current Metrics: Neither account for:
I Pieces misplaced in different puzzles
I Extra pieces from other puzzles
I Goal: Define new quality metrics for Mixed-Bag puzzles
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying Quality33Direct Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Quantifying Solver QualityStandard and Enhanced Direct Accuracy
I Standard Direct Accuracy: Fraction of pieces, c placedin the same location in both the ground-truth and solvedimage versus the total number of pieces, n
DA =cn
(5)
I Enhanced Direct Accuracy Score (EDAS): Modifieddirect accuracy that accounts for missing and extra pieces.
EDASPi = arg maxSj∈S
ci,j
ni +∑
k 6=i (mk,j )(6)
I Direct Accuracy Range: 0 to 1
I Perfectly Reconstructed Image: All pieces are placed intheir original location (DA = EDAS = 1)
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying Quality33Direct Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Quantifying Solver QualityStandard and Enhanced Direct Accuracy
I Standard Direct Accuracy: Fraction of pieces, c placedin the same location in both the ground-truth and solvedimage versus the total number of pieces, n
DA =cn
(5)
I Enhanced Direct Accuracy Score (EDAS): Modifieddirect accuracy that accounts for missing and extra pieces.
EDASPi = arg maxSj∈S
ci,j
ni +∑
k 6=i (mk,j )(6)
I Direct Accuracy Range: 0 to 1
I Perfectly Reconstructed Image: All pieces are placed intheir original location (DA = EDAS = 1)
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying Quality33Direct Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Quantifying Solver QualityStandard and Enhanced Direct Accuracy
I Standard Direct Accuracy: Fraction of pieces, c placedin the same location in both the ground-truth and solvedimage versus the total number of pieces, n
DA =cn
(5)
I Enhanced Direct Accuracy Score (EDAS): Modifieddirect accuracy that accounts for missing and extra pieces.
EDASPi = arg maxSj∈S
ci,j
ni +∑
k 6=i (mk,j )(6)
I Direct Accuracy Range: 0 to 1
I Perfectly Reconstructed Image: All pieces are placed intheir original location (DA = EDAS = 1)
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying Quality34Direct Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Direct AccuracyExample – Effect of Shifts
Problem: Direct accuracy is highly vulnerable to shifts, inparticular when puzzle dimensions are not fixed
Ground-Truth Image Solver Output
Conclusion: Direct accuracy can be overly punitive.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying Quality34Direct Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Direct AccuracyExample – Effect of Shifts
Problem: Direct accuracy is highly vulnerable to shifts, inparticular when puzzle dimensions are not fixed
Ground-Truth Image
Solver Output
Conclusion: Direct accuracy can be overly punitive.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying Quality34Direct Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Direct AccuracyExample – Effect of Shifts
Problem: Direct accuracy is highly vulnerable to shifts, inparticular when puzzle dimensions are not fixed
Ground-Truth Image Solver Output
Conclusion: Direct accuracy can be overly punitive.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying Quality34Direct Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Direct AccuracyExample – Effect of Shifts
Problem: Direct accuracy is highly vulnerable to shifts, inparticular when puzzle dimensions are not fixed
Ground-Truth Image Solver Output
Conclusion: Direct accuracy can be overly punitive.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying Quality35Direct Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Direct AccuracyShiftable Enhanced Direct Accuracy Score (SEDAS)
I Solution: Allow the reference point for direct accuracy toshift beyond the upper left corner of the image
I Shiftable Enhanced Direct Accuracy Score (SEDAS):Select the reference point, l , within radius dmin of the upperleft corner of the solved puzzle
I dmin – Manhattan distance between the upper left corner ofthe solved image and the nearest puzzle piece
I Formal Definition of SEDAS:
SEDASPi = arg maxl∈L
(arg max
Sj∈S
ci,j,l
ni +∑
k 6=i (mk,j )
)(7)
I SEDAS Range: 0 to 1
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying Quality36Direct Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Direct AccuracyExample – Shiftable Reference Point
Solver Output
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying Quality36Direct Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Direct AccuracyExample – Shiftable Reference Point
Solver Output Direct Accuracy Reference Point
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying Quality36Direct Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Direct AccuracyExample – Shiftable Reference Point
Solver Output SEDAS Reference Points
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
37Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Quantifying Solver QualityStandard and Enhanced Neighbor Accuracy
I Standard Neighbor Accuracy: Ratio of puzzle piecesides adjacent in both the original and solved images, a,versus the total number of sides, n · q
NA =a
n · q(8)
I Enhanced Neighbor Accuracy Score (ENAS): Modifiedneighbor accuracy that accounts for missing and extrapieces.
ENASPi = arg maxSj∈S
ai,j
q(ni +∑
k 6=i (mk,j )(9)
I Neighbor Accuracy Range: 0 to 1
I Advantage of Neighbor Accuracy: Less vulnerable toshifts than direct accuracy
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
37Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Quantifying Solver QualityStandard and Enhanced Neighbor Accuracy
I Standard Neighbor Accuracy: Ratio of puzzle piecesides adjacent in both the original and solved images, a,versus the total number of sides, n · q
NA =a
n · q(8)
I Enhanced Neighbor Accuracy Score (ENAS): Modifiedneighbor accuracy that accounts for missing and extrapieces.
ENASPi = arg maxSj∈S
ai,j
q(ni +∑
k 6=i (mk,j )(9)
I Neighbor Accuracy Range: 0 to 1
I Advantage of Neighbor Accuracy: Less vulnerable toshifts than direct accuracy
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
37Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Quantifying Solver QualityStandard and Enhanced Neighbor Accuracy
I Standard Neighbor Accuracy: Ratio of puzzle piecesides adjacent in both the original and solved images, a,versus the total number of sides, n · q
NA =a
n · q(8)
I Enhanced Neighbor Accuracy Score (ENAS): Modifiedneighbor accuracy that accounts for missing and extrapieces.
ENASPi = arg maxSj∈S
ai,j
q(ni +∑
k 6=i (mk,j )(9)
I Neighbor Accuracy Range: 0 to 1
I Advantage of Neighbor Accuracy: Less vulnerable toshifts than direct accuracy
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
37Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Quantifying Solver QualityStandard and Enhanced Neighbor Accuracy
I Standard Neighbor Accuracy: Ratio of puzzle piecesides adjacent in both the original and solved images, a,versus the total number of sides, n · q
NA =a
n · q(8)
I Enhanced Neighbor Accuracy Score (ENAS): Modifiedneighbor accuracy that accounts for missing and extrapieces.
ENASPi = arg maxSj∈S
ai,j
q(ni +∑
k 6=i (mk,j )(9)
I Neighbor Accuracy Range: 0 to 1
I Advantage of Neighbor Accuracy: Less vulnerable toshifts than direct accuracy
Experimental Results
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
38Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Experimental Results
I Paikin & Tal’s algorithm is the current state of the art andwas used as the reference for performance comparison
I Standard Test Conditions:I Puzzle Type: 2I Dimensions Fixed: NoI Piece Width: 28 pixelsI Benchmark: Twenty 805 piece images [6]
I Number of Ground-Truth Inputs: 1 to 5
# Puzzles 1 2 3 4 5
# Iterations 20 55 25 8 5
I Test Condition Variation: Only Paikin & Tal’s algorithmwas provided the number of input puzzles.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
38Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Experimental Results
I Paikin & Tal’s algorithm is the current state of the art andwas used as the reference for performance comparison
I Standard Test Conditions:I Puzzle Type: 2I Dimensions Fixed: NoI Piece Width: 28 pixelsI Benchmark: Twenty 805 piece images [6]
I Number of Ground-Truth Inputs: 1 to 5
# Puzzles 1 2 3 4 5
# Iterations 20 55 25 8 5
I Test Condition Variation: Only Paikin & Tal’s algorithmwas provided the number of input puzzles.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
38Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Experimental Results
I Paikin & Tal’s algorithm is the current state of the art andwas used as the reference for performance comparison
I Standard Test Conditions:I Puzzle Type: 2I Dimensions Fixed: NoI Piece Width: 28 pixelsI Benchmark: Twenty 805 piece images [6]
I Number of Ground-Truth Inputs: 1 to 5
# Puzzles 1 2 3 4 5
# Iterations 20 55 25 8 5
I Test Condition Variation: Only Paikin & Tal’s algorithmwas provided the number of input puzzles.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
38Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Experimental Results
I Paikin & Tal’s algorithm is the current state of the art andwas used as the reference for performance comparison
I Standard Test Conditions:I Puzzle Type: 2I Dimensions Fixed: NoI Piece Width: 28 pixelsI Benchmark: Twenty 805 piece images [6]
I Number of Ground-Truth Inputs: 1 to 5
# Puzzles 1 2 3 4 5
# Iterations 20 55 25 8 5
I Test Condition Variation: Only Paikin & Tal’s algorithmwas provided the number of input puzzles.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
38Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Experimental Results
I Paikin & Tal’s algorithm is the current state of the art andwas used as the reference for performance comparison
I Standard Test Conditions:I Puzzle Type: 2I Dimensions Fixed: NoI Piece Width: 28 pixelsI Benchmark: Twenty 805 piece images [6]
I Number of Ground-Truth Inputs: 1 to 5
# Puzzles 1 2 3 4 5
# Iterations 20 55 25 8 5
I Test Condition Variation: Only Paikin & Tal’s algorithmwas provided the number of input puzzles.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
39Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Experimental ResultsDetermining Input Puzzle Count
I Goal: Measure the Mixed-Bag Solver’s accuracydetermining the number of input puzzles
I Importance – The Mixed-Bag Solver must estimate thisaccurately to provide meaningful outputs.
I Single Puzzle Accuracy – Represents the solver’sperformance ceiling
I Multiple Puzzle Accuracy – A more general estimate ofthe solver’s performance
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
39Experimental ResultsInput Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Experimental ResultsDetermining Input Puzzle Count
I Goal: Measure the Mixed-Bag Solver’s accuracydetermining the number of input puzzles
I Importance – The Mixed-Bag Solver must estimate thisaccurately to provide meaningful outputs.
I Single Puzzle Accuracy – Represents the solver’sperformance ceiling
I Multiple Puzzle Accuracy – A more general estimate ofthe solver’s performance
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental Results40Input Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Determining Input Puzzle CountSingle Input Puzzle Results
I Summary: 17 out of the 20 images were correctlyidentified as a single ground-truth input
I Misclassified Images: 3 out of the 20 imagesmisclassified as if they were two images.
I All three images have large areas with little variation (e.g., ablue sky, smooth water)
I The solver’s poor performance on these puzzles is due tothe assembler as noted in [4]
I Note: 85% (17/20) represents the accuracy ceiling whensolving multiple puzzles.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental Results40Input Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Determining Input Puzzle CountSingle Input Puzzle Results
I Summary: 17 out of the 20 images were correctlyidentified as a single ground-truth input
I Misclassified Images: 3 out of the 20 imagesmisclassified as if they were two images.
I All three images have large areas with little variation (e.g., ablue sky, smooth water)
I The solver’s poor performance on these puzzles is due tothe assembler as noted in [4]
I Note: 85% (17/20) represents the accuracy ceiling whensolving multiple puzzles.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental Results41Input Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Determining Input Puzzle CountVisual Comparison of a Misclassified Image
Perfectly ReconstructedImage (a) Misclassified Image (b)
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental Results42Input Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Determining Input Puzzle CountMultiple Input Puzzles
I Goal: Measure the Mixed-Bag Solver’s accuracydetermining the input puzzle count for multiple images
I Procedure: Randomly select a specified number ofimages (between 2 and 5) from the 20 image data set.
I Input Puzzle Count Error: Difference between the actualnumber of input puzzles and the number determined bythe Mixed-Bag Solver.
I Example: If 3 images were supplied to the solver but itdetermined there were 4, the error would be 1.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental Results42Input Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Determining Input Puzzle CountMultiple Input Puzzles
I Goal: Measure the Mixed-Bag Solver’s accuracydetermining the input puzzle count for multiple images
I Procedure: Randomly select a specified number ofimages (between 2 and 5) from the 20 image data set.
I Input Puzzle Count Error: Difference between the actualnumber of input puzzles and the number determined bythe Mixed-Bag Solver.
I Example: If 3 images were supplied to the solver but itdetermined there were 4, the error would be 1.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental Results43Input Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Determining Input Puzzle CountMultiple Input Puzzles – Results
0 1 2 30
20
40
60
80
100
75
167
2
4448
4 4
50 50
0 0
60
20 20
0
Size of Input Puzzle Count Error
Freq
uenc
y(%
)
Mixed-Bag Solver’s Input Puzzle Count Error Frequency
2 Puzzles 3 Puzzles 4 Puzzles 5 Puzzles
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental Results44Input Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Determining Input Puzzle CountMultiple Input Puzzles – Results Summary
I Overall Accuracy: 65%
I Iterations with Error Greater than One: 8%
I Accuracy did not significantly degrade as the number ofinput puzzles increased.
I Over-Rejection of Cluster Mergers: The Mixed-BagSolver never underestimated the number of input puzzles.
I Performance may be improved by reducing the minimumclustering similarity threshold or minimum segment size
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental Results44Input Puzzle Count
Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Determining Input Puzzle CountMultiple Input Puzzles – Results Summary
I Overall Accuracy: 65%
I Iterations with Error Greater than One: 8%
I Accuracy did not significantly degrade as the number ofinput puzzles increased.
I Over-Rejection of Cluster Mergers: The Mixed-BagSolver never underestimated the number of input puzzles.
I Performance may be improved by reducing the minimumclustering similarity threshold or minimum segment size
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
45Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Experimental ResultsPerformance on Multiple Input Puzzles
I Goal: Compare the performance of the Mixed-Bag Solver(MBS) and Paikin & Tal’s algorithm
I Procedure: Randomly select a specified number ofimages and input them into both solvers.
I Quality Metrics Used:I Shiftable Enhanced Direct Accuracy Score (SEDAS)I Enhanced Neighbor Accuracy Score (ENAS)I Perfect Reconstruction Percentage
I Note: The results include the Mixed-Bag Solver’sperformance when it correctly estimated the puzzle count.
I This represents the performance ceiling for optimalhierarchical clustering.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
45Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Experimental ResultsPerformance on Multiple Input Puzzles
I Goal: Compare the performance of the Mixed-Bag Solver(MBS) and Paikin & Tal’s algorithm
I Procedure: Randomly select a specified number ofimages and input them into both solvers.
I Quality Metrics Used:I Shiftable Enhanced Direct Accuracy Score (SEDAS)I Enhanced Neighbor Accuracy Score (ENAS)I Perfect Reconstruction Percentage
I Note: The results include the Mixed-Bag Solver’sperformance when it correctly estimated the puzzle count.
I This represents the performance ceiling for optimalhierarchical clustering.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
46Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Performance on Multiple Input PuzzlesShiftable Enhanced Direct Accuracy Score (SEDAS)
2 3 4 50
0.2
0.4
0.6
0.8
1
Number of Input Puzzles
SE
DA
S
Effect of the Number of Input Puzzles on SEDAS
MBS Correct Puzzle Count MBS All Paikin & Tal
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
47Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Performance on Multiple Input PuzzlesEnhanced Neighbor Accuracy Score (ENAS)
2 3 4 50
0.2
0.4
0.6
0.8
1
Number of Input Puzzles
EN
AS
Effect of the Number of Input Puzzles on ENAS
MBS Correct Puzzle Count MBS All Paikin & Tal
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
48Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Performance on Multiple Input PuzzlesPerfect Reconstruction Percentage
2 3 4 50
5
10
15
20
25
30
Number of Input Puzzles
Per
fect
Rec
onst
ruct
ion
(%)
Effect of the Number of Input Puzzles on Perfect Reconstruction
MBS Correct Puzzle Count MBS All Paikin & Tal
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
49Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Performance on Multiple Input PuzzlesResults Summary
I Summary: The Mixed-Bag Solver significantlyoutperformed Paikin & Tal’s algorithm across all metrics.
I This is notwithstanding that only their algorithm wassupplied with the number of input puzzles.
I Puzzle Input Count: Unlike Paikin & Tal’s algorithm, theMixed-Bag Solver saw no significant decrease inperformance with additional input puzzles
I Effect of Clustering Errors: Performance only decreasedslightly when incorrectly estimated input puzzle count.
I Many of the extra puzzles were relatively insignificant in size
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
49Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Performance on Multiple Input PuzzlesResults Summary
I Summary: The Mixed-Bag Solver significantlyoutperformed Paikin & Tal’s algorithm across all metrics.
I This is notwithstanding that only their algorithm wassupplied with the number of input puzzles.
I Puzzle Input Count: Unlike Paikin & Tal’s algorithm, theMixed-Bag Solver saw no significant decrease inperformance with additional input puzzles
I Effect of Clustering Errors: Performance only decreasedslightly when incorrectly estimated input puzzle count.
I Many of the extra puzzles were relatively insignificant in size
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
49Solver Comparison
Conclusions
Dept. of Computer ScienceSan José State University
Performance on Multiple Input PuzzlesResults Summary
I Summary: The Mixed-Bag Solver significantlyoutperformed Paikin & Tal’s algorithm across all metrics.
I This is notwithstanding that only their algorithm wassupplied with the number of input puzzles.
I Puzzle Input Count: Unlike Paikin & Tal’s algorithm, theMixed-Bag Solver saw no significant decrease inperformance with additional input puzzles
I Effect of Clustering Errors: Performance only decreasedslightly when incorrectly estimated input puzzle count.
I Many of the extra puzzles were relatively insignificant in size
Conclusions
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
50Conclusions
Dept. of Computer ScienceSan José State University
Conclusions
I This thesis presented a fully-automated solver forMixed-Bag puzzles.
I Mixed-Bag Solver significantly outperforms the currentstate of the art while receiving no externally suppliedinformation.
I Introduced the first set of solver quality metrics forMixed-Bag puzzles.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
50Conclusions
Dept. of Computer ScienceSan José State University
Conclusions
I This thesis presented a fully-automated solver forMixed-Bag puzzles.
I Mixed-Bag Solver significantly outperforms the currentstate of the art while receiving no externally suppliedinformation.
I Introduced the first set of solver quality metrics forMixed-Bag puzzles.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Thesis Goals
IntroductionPrevious Work
Mixed-Bag SolverAssembler
Segmentation
Stitching
Hierarchical Clustering
Final Seed Piece Selection
Final Assembly
Quantifying QualityDirect Accuracy
Neighbor Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
51Conclusions
Dept. of Computer ScienceSan José State University
Summary of Topics
Thesis GoalsIntroduction
Previous WorkMixed-Bag Solver
AssemblerSegmentationStitchingHierarchical ClusteringFinal Seed Piece SelectionFinal Assembly
Quantifying QualityDirect AccuracyNeighbor Accuracy
Experimental ResultsInput Puzzle CountSolver Comparison
Conclusions
Appendix
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Conclusions53Future Work
IntroductionPuzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
ConclusionsFuture Work
I Improved AssemblerI Prioritize placement using multiple best buddies
I Address placement performance in regions with low bestbuddy density
I Dynamic determination of the segment clusteringthreshold
I Expanded stitching piece selection
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Conclusions53Future Work
IntroductionPuzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
ConclusionsFuture Work
I Improved AssemblerI Prioritize placement using multiple best buddies
I Address placement performance in regions with low bestbuddy density
I Dynamic determination of the segment clusteringthreshold
I Expanded stitching piece selection
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Conclusions53Future Work
IntroductionPuzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
ConclusionsFuture Work
I Improved AssemblerI Prioritize placement using multiple best buddies
I Address placement performance in regions with low bestbuddy density
I Dynamic determination of the segment clusteringthreshold
I Expanded stitching piece selection
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
Conclusions54Future Work
IntroductionPuzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
List of References I
[1] T. Altman, “Solving the jigsaw puzzle problem in linear time,” Applied Artificial Intelligence, vol. 3, pp. 453–462, Jan. 1990.
[2] E. D. Demaine and M. L. Demaine, “Jigsaw puzzles, edge matching, and polyomino packing: Connections andcomplexity,” Graphs and Combinatorics, vol. 23 (Supplement), pp. 195–208, June 2007.
[3] A. C. Gallagher, “Jigsaw puzzles with pieces of unknown orientation,” in Proceedings of the 2012 IEEE Conference onComputer Vision and Pattern Recognition (CVPR), CVPR ’12, pp. 382–389, IEEE Computer Society, 2012.
[4] G. Paikin and A. Tal, “Solving multiple square jigsaw puzzles with missing pieces,” in Proceedings of the 2015 IEEEConference on Computer Vision and Pattern Recognition (CVPR), CVPR ’15, IEEE Computer Society, 2015.
[5] Z. Hammoudeh, “Github - puzzle solver thesis repository.” https://github.com/ZaydH/Thesis.(Accessed on 10/23/2016).
[6] D. Pomeranz, M. Shemesh, and O. Ben-Shahar, “Computational jigsaw puzzle solving.”https://www.cs.bgu.ac.il/~icvl/icvl_projects/automatic-jigsaw-puzzle-solving/, 2011.(Accessed on 05/01/2016).
[7] A. Olmos and F. A. A. Kingdom, “McGill calibrated colour image database.” http://tabby.vision.mcgill.ca/, 2005.(Accessed on 05/01/2016).
[8] P.-N. Tan, M. Steinbach, and V. Kumar, Introduction to Data Mining, (First Edition).Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., 2005.
[9] T. S. Cho, S. Avidan, and W. T. Freeman, “A probabilistic image jigsaw puzzle solver,” in Proceedings of the 2010 IEEEConference on Computer Vision and Pattern Recognition (CVPR), CVPR ’10, pp. 183–190, IEEE Computer Society,2010.
[10] D. Pomeranz, M. Shemesh, and O. Ben-Shahar, “A fully automated greedy square jigsaw puzzle solver,” in Proceedingsof the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR ’11, pp. 9–16, IEEEComputer Society, 2011.
[11] H. Braxmeier and S. Steinberger, “Pixabay.” https://pixabay.com/.(Accessed on 05/15/2016).
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
Introduction55Puzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzle Types
There are four primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.
I Type 1: Puzzle dimensions and piece rotation are known.May have “anchor” piece(s) fixed in the correct location(s).
I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.
I Type 3: All piece locations are known. Only rotation isunknown.
I Mixed-Bag: Pieces come from multiple puzzles
Mixed-Bag puzzles are the focus of this thesis.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
Introduction55Puzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzle Types
There are four primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.
I Type 1: Puzzle dimensions and piece rotation are known.May have “anchor” piece(s) fixed in the correct location(s).
I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.
I Type 3: All piece locations are known. Only rotation isunknown.
I Mixed-Bag: Pieces come from multiple puzzles
Mixed-Bag puzzles are the focus of this thesis.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
Introduction55Puzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzle Types
There are four primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.
I Type 1: Puzzle dimensions and piece rotation are known.May have “anchor” piece(s) fixed in the correct location(s).
I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.
I Type 3: All piece locations are known. Only rotation isunknown.
I Mixed-Bag: Pieces come from multiple puzzles
Mixed-Bag puzzles are the focus of this thesis.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
Introduction55Puzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzle Types
There are four primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.
I Type 1: Puzzle dimensions and piece rotation are known.May have “anchor” piece(s) fixed in the correct location(s).
I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.
I Type 3: All piece locations are known. Only rotation isunknown.
I Mixed-Bag: Pieces come from multiple puzzles
Mixed-Bag puzzles are the focus of this thesis.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
Introduction55Puzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzle Types
There are four primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.
I Type 1: Puzzle dimensions and piece rotation are known.May have “anchor” piece(s) fixed in the correct location(s).
I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.
I Type 3: All piece locations are known. Only rotation isunknown.
I Mixed-Bag: Pieces come from multiple puzzles
Mixed-Bag puzzles are the focus of this thesis.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
Introduction55Puzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
IntroductionJig Swap Puzzle Types
There are four primary jig swap puzzle types as formalizedby [3]. In all cases, the “ground-truth” input is unknown.
I Type 1: Puzzle dimensions and piece rotation are known.May have “anchor” piece(s) fixed in the correct location(s).
I Type 2: All piece locations and rotations unknown. Puzzledimensions may be known.
I Type 3: All piece locations are known. Only rotation isunknown.
I Mixed-Bag: Pieces come from multiple puzzles
Mixed-Bag puzzles are the focus of this thesis.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
56Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
Previous Work
I Cho et al. [9] – Introduced the first modern jig swap puzzlesolver
I Graphical model-based Type 1 solver
I Puzzle dimensions are known
I Used one or more anchor pieces
I Defined quality metrics for Type 1 and Type 2 puzzles
I Established the standard comparative test conditions
I Pomeranz et al. [10] – Iterative, greedy Type 1 puzzlesolver
I Eliminated the use of anchor pieces
I Created multiple solver benchmarks of various sizes
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
56Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
Previous Work
I Cho et al. [9] – Introduced the first modern jig swap puzzlesolver
I Graphical model-based Type 1 solver
I Puzzle dimensions are known
I Used one or more anchor pieces
I Defined quality metrics for Type 1 and Type 2 puzzles
I Established the standard comparative test conditions
I Pomeranz et al. [10] – Iterative, greedy Type 1 puzzlesolver
I Eliminated the use of anchor pieces
I Created multiple solver benchmarks of various sizes
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
Previous Work
57Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
IntroductionBest Buddies
I Basis of all Modern Jig Swap Solvers: The morecompatible two pieces are, the more likely they are to beadjacent.
I Best Buddies: A pair of puzzles pieces that are morecompatible with each other on their respective sides thanthey are to any other piece [10]
I Note: Not all puzzle pieces will have a best buddy.
∀pk∀sz ,C(pi , sx ,pj , sy ) ≥ C(pi , sx ,pk , sz)
and
∀pk∀sz ,C(pj , sy ,pi , sx ) ≥ C(pj , sy ,pk , sz)
(10)
I Importance of Best Buddies: Key adjacency indicator
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
Previous Work
Best Buddies58Best Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
Quantifying Solver QualityBest Buddy Density
I Best Buddy Density (BBD): A metric for quantifying thebest buddy profile of an image that is independent ofimage size.
BBD =b
n · q(11)
I A greater BBD means the pieces are more differentiatedmaking the puzzle easier to solve.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
Previous Work
Best Buddies59Best Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
Best Buddy DensityVisualization
Visualizing Best Buddy DensityI Transform each puzzle piece into a square consisting of
four isosceles triangles.
I Color each triangle according to whether the adjacentpiece is a best buddy. The scheme used in this thesis:
No BestBuddy
Non-AdjacentBest Buddy
AdjacentBest Buddy
No PiecePresent
I Areas with higher best buddy density will have more greentriangles.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
Previous Work
Best Buddies60Best Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
Best Buddy DensityVisualization Example
Original Image [11]
Best Buddy Visualization
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
Previous Work
Best Buddies60Best Buddy Density
Experimental ResultsSingle Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
Best Buddy DensityVisualization Example
Original Image [11] Best Buddy Visualization
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental Results61Single Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
Determining Input Puzzle CountComparison of Best Buddy Density for Misclassified Images
Perfectly ReconstructedImage (a)
Best Buddy Visualization (a)
Misclassified Image (b)
Best Buddy Visualization (b)
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental Results61Single Input Puzzle
Ten Puzzle Results
Dept. of Computer ScienceSan José State University
Determining Input Puzzle CountComparison of Best Buddy Density for Misclassified Images
Perfectly ReconstructedImage (a) Best Buddy Visualization (a)
Misclassified Image (b) Best Buddy Visualization (b)
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
62Ten Puzzle Results
Dept. of Computer ScienceSan José State University
Experimental ResultsSolving More than Five Puzzles
I As the number of puzzles increases, the difficulty ofsimultaneously reconstructing them also increases.
I Current State of the Art: Paikin & Tal [4] solved up to fivepuzzles simultaneously.
I Goal: Compare the performance of the Mixed-Bag Solverand Paikin & Tal’s algorithm on 10 puzzles.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
62Ten Puzzle Results
Dept. of Computer ScienceSan José State University
Experimental ResultsSolving More than Five Puzzles
I As the number of puzzles increases, the difficulty ofsimultaneously reconstructing them also increases.
I Current State of the Art: Paikin & Tal [4] solved up to fivepuzzles simultaneously.
I Goal: Compare the performance of the Mixed-Bag Solverand Paikin & Tal’s algorithm on 10 puzzles.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
62Ten Puzzle Results
Dept. of Computer ScienceSan José State University
Experimental ResultsSolving More than Five Puzzles
I As the number of puzzles increases, the difficulty ofsimultaneously reconstructing them also increases.
I Current State of the Art: Paikin & Tal [4] solved up to fivepuzzles simultaneously.
I Goal: Compare the performance of the Mixed-Bag Solverand Paikin & Tal’s algorithm on 10 puzzles.
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
63Ten Puzzle Results
Dept. of Computer ScienceSan José State University
Ten Puzzle ResultsSummary
I Paikin & TalI Seed of nine images came from just three input images
I SEDAS and EDAS greater than 0.9 for only one image
I No perfectly reconstructed images
I Mixed-Bag SolverI SEDAS and EDAS greater than 0.9 for all images
I Four images perfectly reconstructed
I Results comparable to Paikin & Tal’s algorithm solving eachpuzzle individually
51
A Fully-AutomatedSolver for Multiple
Square JigsawPuzzles Using
Hierarchical Clustering
ConclusionsFuture Work
IntroductionPuzzle Types
Previous Work
Best BuddiesBest Buddy Density
Experimental ResultsSingle Input Puzzle
63Ten Puzzle Results
Dept. of Computer ScienceSan José State University
Ten Puzzle ResultsSummary
I Paikin & TalI Seed of nine images came from just three input images
I SEDAS and EDAS greater than 0.9 for only one image
I No perfectly reconstructed images
I Mixed-Bag SolverI SEDAS and EDAS greater than 0.9 for all images
I Four images perfectly reconstructed
I Results comparable to Paikin & Tal’s algorithm solving eachpuzzle individually