Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by:...
-
date post
21-Dec-2015 -
Category
Documents
-
view
219 -
download
2
Transcript of Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by:...
![Page 1: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/1.jpg)
Simultaneous Structure and Texture Image Inpaintingby: Bertalmio, Sapiro, Vese, Osher
Presented by: Shane Brennan
June 7, 2007EE 264 – Spring 2007
![Page 2: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/2.jpg)
What Is Inpainting?
![Page 3: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/3.jpg)
![Page 4: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/4.jpg)
Inpainting can also be used for other purposes including:
Humor Entertainment Improving aesthetic quality of images
And for less playful purposes…
![Page 5: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/5.jpg)
![Page 6: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/6.jpg)
How To Inpaint (yes you can do
it too!)
Identify the regions you want to fill/remove Continue any lines arriving at those regions Fill in the regions with texture/color from the
surrounding areas
Enjoy!
![Page 7: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/7.jpg)
Image Inpaintingby: Marcelo Bertalmio and Guillermo Sapiro
Proceedings of SIGGRAPH 2000
![Page 8: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/8.jpg)
The (very) Basic Idea
Form an iterative algorithm
Each update gets you closer to the desired result
![Page 9: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/9.jpg)
The Update Image
Want to propagate “information” (aka lines) into the region being inpainted
Need information, and direction it is heading
Project information on that direction
Rate of Change of Information
Direction of Information
![Page 10: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/10.jpg)
What Is Information?
Since want result to be smooth, use laplacian of the image to represent information.L = I * 2
The rate of change of the information entering a pixel is the derivative of the laplacian
= [Li,j+1- Li,j-1, Li+1,j – Li-1,j]
![Page 11: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/11.jpg)
Propagation Direction
Tangent to the isophate line (which is the information) arriving at the boundary
Tangent easily computed as:
N = [-IY, Ix]
note: I use the (x,y) coordinate notation
![Page 12: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/12.jpg)
Anisotropic Diffusion Areas where N = [0, 0] will never get updated! These
are smooth regions “Bleed” smooth regions into the region being inpainted Also ensures lines stay smooth and curved BUT! Very problematic! My form of diffusion:
The authors form: ???
![Page 13: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/13.jpg)
![Page 14: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/14.jpg)
Bad Result Due to My Diffusion
![Page 15: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/15.jpg)
How About Texture?
![Page 16: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/16.jpg)
Texture Synthesis by Non-Parametric Samplingby: Alexei Efros and Thomas Leung
IEEE International Conference on Computer Vision, Corfu, Greece, September 1999
![Page 17: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/17.jpg)
![Page 18: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/18.jpg)
How It Works
Take a pixel to be synthesized. Find which pixels near it have already been synthesized (or pre-existed). Define this to be the mask
For every pixel in the image (the candidates), compare the WxW neighborhood to the neighborhood around the pixel to be synthesized using a distance metric, but only on pixels defined by the mask
Keep either the K most similar neighborhoods, or the neighborhoods whose distance is less than T (W, K, and T are user-defined values)
Of the remaining regions, select one at random Assign the intensity of the center pixel of the selected region to
the pixel being synthesized
![Page 19: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/19.jpg)
A Visualization…
![Page 20: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/20.jpg)
![Page 21: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/21.jpg)
How About Structure?
![Page 22: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/22.jpg)
The Problem
Image inpainting works well on regions with structure, but not on regions with texture
Texture synthesis works well on regions with texture, but not on regions with structure
Early papers: for each pixel decide if structure or texture, and perform the appropriate filling method
But there is a better way…
![Page 23: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/23.jpg)
Simultaneous Structure and Texture Image Inpaintingby: Bertalmio, Sapiro, Vese, Osher
IEEE Transactions On Image Processing, Vol. 12, No. 8, August 2003
![Page 24: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/24.jpg)
How It Works
Decompose image into two parts: structure image and texture image
Perform inpainting on the structure image
Perform texture synthesis on the texture image
Recombine the two images to get final result
But how to decompose? no, not like that
![Page 25: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/25.jpg)
Modeling Textures With Total Variation Minimization and Oscillating Patterns in Image Processing
by: Stanley Osher and Luminita Vese
Journal of Scientific Computing, Vol. 19, Nos. 1–3, December 2003
![Page 26: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/26.jpg)
The Initial Version
Considered an image to be some underlying “real image” and then noise added. Want to remove the noise (Rudin, Ohser, and Fatemi. 1992)
Find structure image, u, that minimizes:
Smoothness Term Data Fidelity
![Page 27: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/27.jpg)
Incorporating the Texture
No texture in there! But lets consider image to be f = u + v, where u is structure and v is texture. Note v = f – u
Solutions to this are known! (refer to report for some references on the solution)
![Page 28: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/28.jpg)
Cutting to the Chase
Model texture as x and y components, call them g1 and g2
Some magic... (refer to the paper for more)
Smoothness on u Reconstruction Error Smoothness on v (remove noise)
![Page 29: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/29.jpg)
An Iterative Solution
Break the optimization down into an iterative solution. Although more iterations aren’t always better, in practice need to play with # of iterations
Also need to play with tuning parameters. Though there are default values which work well in a broad array of images
![Page 30: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/30.jpg)
Original Image (f)
Structure Image (u) Texture Image (v)
![Page 31: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/31.jpg)
Original Image (f) Structure Image (u) Texture Image (v)
![Page 32: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/32.jpg)
Back to Inpainting!
![Page 33: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/33.jpg)
The Algorithm
Decompose Iin into sub-images u and v Perform inpainting on u to obtain U Perform texture synthesis on v to obtain V Iout = U + V
And now for some results…
![Page 34: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/34.jpg)
Original Image Structure Image Texture Image
Completed Image Inpainted Structure Image Synthesized Texture Image
![Page 35: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/35.jpg)
Original Image Structure Image Texture Image
Completed Image Inpainted Structure Image Synthesized Texture Image
![Page 36: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/36.jpg)
Original Image Structure Image Texture Image
Completed Image Inpainted Structure Image Synthesized Texture Image
![Page 37: Simultaneous Structure and Texture Image Inpainting by: Bertalmio, Sapiro, Vese, Osher Presented by: Shane Brennan June 7, 2007 EE 264 – Spring 2007.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d585503460f94a37bfc/html5/thumbnails/37.jpg)
References
M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester. Image inpainting. 2000.
A. A. Efros and T. K. Leung. Texture synthesis by non-parametric sampling. In ICCV (2), pages1033–1038, 1999.
M. Bertalmio, L. Vese, G. Sapiro, and S. Osher. Simultaneous structure and texture image inpainting, 2002.
L. Vese and S. Osher. Modeling textures with total variation minimization and oscillating patterns in image processing, 2002.