Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Junhwan Kim and Fabio Pellacini Cornell...

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini

Junhwan Kim and Fabio PellaciniCornell University


GoalGoalGoalGoal

Allow tile deformationAllow tile deformationArbitrarily-shaped container and tiles Arbitrarily-shaped container and tiles Allow tile deformationAllow tile deformationArbitrarily-shaped container and tiles Arbitrarily-shaped container and tiles

1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions


Algorithm CriteriaAlgorithm Criteria

Visually pleasing resultVisually pleasing result

Acceptable computational costAcceptable computational cost

Little user effortLittle user effort

Algorithm CriteriaAlgorithm Criteria

Visually pleasing resultVisually pleasing result

Acceptable computational costAcceptable computational cost


GoalGoalGoalGoal



MotivationMotivation

Related WorkRelated Work

Preparing InputPreparing Input

FrameworkFramework

Basic Mosaic AlgorithmBasic Mosaic Algorithm

Algorithm OptimizationAlgorithm Optimization

ResultsResults

ConclusionsConclusions




FrameworkFramework



ResultsResults


Outline



Giuseppe ArcimboldoGiuseppe Arcimboldo1527-15931527-1593

Giuseppe ArcimboldoGiuseppe Arcimboldo1527-15931527-1593

MotivationFruit FaceMotivationFruit Face



Motivation Fruit Face





FrameworkFramework



ResultsResults





FrameworkFramework



ResultsResults


Outline



Related worksRelated worksRelated worksRelated works


Mosaicing algorithmsMosaicing algorithms

Photomosaics [Silvers,R and Hawley,M]Photomosaics [Silvers,R and Hawley,M]

(1997)(1997)

Simulated Decorative Mosaics [Hausner]Simulated Decorative Mosaics [Hausner]

(siggraph2001)(siggraph2001)

Escherization [Kaplan and Salesin]Escherization [Kaplan and Salesin]


Packing algorithmsPacking algorithms Translational Polygon Containment and Minimal Translational Polygon Containment and Minimal

Enclosure using Mathematical Programming-Enclosure using Mathematical Programming-[Milenkovic] (1999)[Milenkovic] (1999)

Mosaicing algorithmsMosaicing algorithms

Photomosaics [Silvers,R and Hawley,M]Photomosaics [Silvers,R and Hawley,M]

(1997)(1997)

Simulated Decorative Mosaics [Hausner]Simulated Decorative Mosaics [Hausner]


Escherization [Kaplan and Salesin]Escherization [Kaplan and Salesin]


Packing algorithmsPacking algorithms Translational Polygon Containment and Minimal Translational Polygon Containment and Minimal

Enclosure using Mathematical Programming-Enclosure using Mathematical Programming-[Milenkovic] (1999)[Milenkovic] (1999)


Related worksRelated works





FrameworkFramework



ResultsResults





FrameworkFramework



ResultsResults


Outline



Preparing Inputs

• Input=(container image + a set of tiles) of Input=(container image + a set of tiles) of arbitrary shapearbitrary shape

• Since we require a fairly high number of tiles, we need to be able to extract the shape of the tiles directly from the images themselves. We do so using active contours (Snake).

• Active contours are also used to extract the container shape.


Preparing Inputs

• Following

Hausner (Simulated Decorative Mosaics ‘01) approach, we segment the input image to generate a set of disjoint arbitrarily-shaped containers.

• Since we preserve the edge of each container, the final composite will preserve the important edges on the input image.





FrameworkFramework



ResultsResults





FrameworkFramework



ResultsResults


Outline



FrameworkFrameworkFrameworkFramework

Energy minimizationEnergy minimizationEnergy minimizationEnergy minimization

wwCC•E•ECCwwCC•E•ECC


Trying 3 rd tile… Trying 4 th tile…Container

Container to be filled

Container filled

Tiles

Legend

Trying 5 th tile…Trying 2 nd tile…

… high gap energy discard

Gap region

…high overlap energy discard

Overlap region

…high deformationenergy discard

Shape mismatch

…lowest energy Accept

Accepted tile

Trying 1 st tile…

Color mismatch

…high color energy discard

+ w+ wGG•E•EGG+ w+ wGG•E•EGGE E ==

E E ==

+ w+ wOO•E•EOO+ w+ wOO•E•EOO+ w+ wDD•E•EDD+ w+ wDD•E•EDD


Framework (Energy terms Framework (Energy terms evaluation)evaluation)• Ec : Ec : Which is estimated by taking the average of Which is estimated by taking the average of L^2L^2

difference of the colors of the difference of the colors of the final imagefinal image and the and the input input containercontainer at random locations on the surface of the container . at random locations on the surface of the container .

• EEGG,E,EO O : : Each vertex of a tile is attached with a Each vertex of a tile is attached with a springspring to the to the nearestnearest edge of the other tiles or the container. If the edge of the other tiles or the container. If the signed signed distance ddistance d between the vertex and the anchor is positive (a gap between the vertex and the anchor is positive (a gap between them), we add d^2/2 to Ebetween them), we add d^2/2 to EGG. On the other hand, if d is . On the other hand, if d is negative, we add d^2/2 to Eo.negative, we add d^2/2 to Eo.

• EEDD: :


Advantages of Energy minimizationAdvantages of Energy minimization

Easy to controlEasy to control

Easy to extendEasy to extend

PhotomosaicPhotomosaic

Simulated Decorative MosaicSimulated Decorative Mosaic


Advantages of Energy minimizationAdvantages of Energy minimization

Easy to controlEasy to control

Easy to extendEasy to extend

PhotomosaicPhotomosaic

Simulated Decorative MosaicSimulated Decorative Mosaic


FrameworkFrameworkFrameworkFramework






FrameworkFramework



ResultsResults





FrameworkFramework



ResultsResults


Outline



Basic AlgorithmBasic AlgorithmOverviewOverviewBasic AlgorithmBasic AlgorithmOverviewOverview


a) Initial containerimage

b) Tile contours aftertile placement

c) Tile contours aftertile refinement

d) Final JigsawImage Mosaic

Phase 3:Adjusting images

Phase 1:Placing tiles

Phase 2:Refining tiles

In the third phase, we assemble the final mosaic by placing each tile in its position and warping each image to its final deformed shape using the image warping technique presented in (image warping by RBF,1994, Arad,N)


E = wE = wCC•E•ECC + w + wGG•E•EGG + w + wOO•E•EOO + w + wDD•E•EDD E = wE = wCC•E•ECC + w + wGG•E•EGG + w + wOO•E•EOO + w + wDD•E•EDD E = wE = wCC•E•ECC + w + wGG•E•EGG + w + wOO•E•EOO + w+ wDD•E•EDD E = wE = wCC•E•ECC + w + wGG•E•EGG + w + wOO•E•EOO + w+ wDD•E•EDD

Packing Packing Packing Packing

Basic AlgorithmBasic AlgorithmPackingPackingBasic AlgorithmBasic AlgorithmPackingPacking

Tile Container withplaced tile

Container fornext iteration

a b cInitial

container



Basic AlgorithmBasic AlgorithmPackingPackingBasic AlgorithmBasic AlgorithmPackingPacking

Container Available Tiles

Place 1st tile Cannot place next

Try again 1st tile

Backtrack

Place next Done

a b

c



Placing a tile

• When placing a tile in a container of arbitrary shape, it would be prohibitive to try every possible location.

• In order to reduce the branching overhead , we try those locations that are most likely to make the container shape easier to fill after updating.

• Unfortunately this depends on which tile we place.

• Nevertheless, we can guess how the container would look after we put an “average” tile. A container will be easier to fill if it does not have a protrusion and is as convex as possible.


Placing a tile

• Before placing a new tile, we construct a CVD (centroidal voronoi diagram), where each site has an area roughly equal to the average size of tiles.

• We then select a random site among the ones that have the least number of neighbors, thus making the container as easy as possible to fill.


The refinement phase of our approach solves this issue by deforming the tiles obtained from the packing stage, while balancing between maintaining the original tile shape as closely as possible and minimizing the full energy function.

The refinement phase of our approach solves this issue by deforming the tiles obtained from the packing stage, while balancing between maintaining the original tile shape as closely as possible and minimizing the full energy function.

E = wE = wCC•E•ECC + w + wGG•E•EGG + + wwOO•E•EOO


PackingPacking PackingPacking

Refinement Refinement Refinement Refinement

Basic AlgorithmBasic AlgorithmRefinementRefinementBasic AlgorithmBasic AlgorithmRefinementRefinement

+ + wwDD•E•E

DD

+ + wwDD•E•E

DD



Basic AlgorithmBasic AlgorithmRefinementRefinementBasic AlgorithmBasic AlgorithmRefinementRefinement

a) Initial contours

b) Intermediate contours

c) Converged contours






FrameworkFramework



ResultsResults





FrameworkFramework



ResultsResults


Outline



Algorithm OptimizationAlgorithm Optimization Algorithm OptimizationAlgorithm Optimization

Placing a tilePlacing a tile

Branch-bound with Look-aheadBranch-bound with Look-ahead

Geometric hashingGeometric hashing

Placing a tilePlacing a tile

Branch-bound with Look-aheadBranch-bound with Look-ahead

Geometric hashingGeometric hashing


Vtile(Vcontainer): the number of vertices per tile (container)

Ntile: the number of tiles in the database

NtileInContainer: the number of tiles in the container

b is the overhead due to branching in the search tree (backtracking).


CVD connected graph

CVD cells

Selected position:only two neighbors in the CVD graph

Legend

Algorithm OptimizationAlgorithm OptimizationPlacing a tilePlacing a tileAlgorithm OptimizationAlgorithm OptimizationPlacing a tilePlacing a tile



Algorithm OptimizationAlgorithm Optimization Look ahead Look ahead (Russell and Norvig 1994)(Russell and Norvig 1994)

• Every time we cannot find a suitable tile to fill a container, we need to backtrack to the configuration that has minimal energy so far.

• To reduce branching overhead, we use a look-ahead technique.

• When placing a new tile, we penalize tiles that will make it harder to fill the container in the next iteration.

• To do this we add a term to the energy formulation that takes into account how the container will look after tile placement.

• The container shape term advocates for containers with a small area and short circumference.




Packing Packing Packing Packing

EELALA = w = wAA••areaareaEELALA = w = wAA••areaarea

Look ahead Look ahead Look ahead Look ahead

Algorithm OptimizationAlgorithm OptimizationLook ahead Look ahead (Russell and Norvig (Russell and Norvig 1994)1994)

Algorithm OptimizationAlgorithm OptimizationLook ahead Look ahead (Russell and Norvig (Russell and Norvig 1994)1994)

+ + wwLALA•E•ELL

AA

+ + wwLALA•E•ELL

AA

+ (1 - w+ (1 - wAA) ) ••lengthlength22

+ (1 - w+ (1 - wAA) ) ••lengthlength22


Where area is the container’s area, length is its boundary length, and WA controls the weight of area in relation to the weight of length .


Algorithm OptimizationAlgorithm Optimization Container cleanup Container cleanup• After we place a tile in a container, we update the

container by subtracting the tile from the container.• However, the new container can be very jagged, or even

have disjoint regions.• If these fragments are shallower than the shallowest tile,

we know it can never be filled with any existing tile. In that case, it is safe to separate those fragments and consider them as a gap.

• This cleanup process reduces the running time by cutting the number of vertices in the container. It also reduces the branching overhead, since it prevents the algorithm from wasting time attempting to fill unpromising fragments of the container.


Algorithm OptimizationAlgorithm Optimization Geometric Hashing Geometric Hashing

• We use geometric hashing to select a few tiles that will suit to a particular position in the container. We then evaluate the energy term for them and pick the best fitting one.

• Intuitively we use geometric hashing as a pruning technique to reject bad tiles.

• We create a grid of squares in the plane in a preprocessing phase. Each square corresponds to a hash table entry.

• If a shape boundary crosses a square, we will record the tile ID and its orientation as an entry in the list attached to that hash table entry.


Algorithm OptimizationAlgorithm Optimization Geometric Hashing Geometric Hashing

• Every time we need to place a new tile in a specific position in the container during the packing stage we register the container boundary segment to the hash table and access the hash table entries of the squares that the container passes through; for every entry found there, we cast a vote for the (tile ID, tile orientation) pair.


a) Bad tile: 15 votes b) Good tile: 22 votesLegend

Container contour

Tile contour

Container and tile contour overlap: cast a vote

Algorithm OptimizationAlgorithm OptimizationGeometric HashingGeometric HashingAlgorithm OptimizationAlgorithm OptimizationGeometric HashingGeometric Hashing[Wolfson and Rigoutsos, 1997][Wolfson and Rigoutsos, 1997][Wolfson and Rigoutsos, 1997][Wolfson and Rigoutsos, 1997]






FrameworkFramework



ResultsResults





FrameworkFramework



ResultsResults


Outline



ResultsResultsResultsResults




a) Base case b) Lower color weight

c) Lower overlap weight

d) Tiles of finer scales






FrameworkFramework



ResultsResults





FrameworkFramework



ResultsResults


Outline



ContributionsContributionsNew kind of mosaicNew kind of mosaic

General framework for mosaicsGeneral framework for mosaics

Effective algorithmEffective algorithm

Future workFuture work

Faster algorithmFaster algorithm

Extension to 3DExtension to 3D

ContributionsContributionsNew kind of mosaicNew kind of mosaic

General framework for mosaicsGeneral framework for mosaics

Effective algorithmEffective algorithm

Future workFuture work

Faster algorithmFaster algorithm

Extension to 3DExtension to 3D

ConclusionsConclusionsConclusionsConclusions



報告完畢

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Junhwan Kim and Fabio Pellacini Cornell...

Documents

Transcript of Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Junhwan Kim and Fabio Pellacini Cornell...