Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler,...
-
Upload
pauline-perkins -
Category
Documents
-
view
212 -
download
0
Transcript of Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler,...
![Page 1: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/1.jpg)
Accurate Binary Image SelectionFrom Inaccurate User Input
Kartic Subr, Sylvain Paris, Cyril Soler, Jan KautzUniversity College London, Adobe Research, INRIA-Grenoble
![Page 2: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/2.jpg)
Selection is a common operation in images
![Page 3: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/3.jpg)
For example
![Page 4: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/4.jpg)
Tools available: Related work
• Simple brush and lasso– magnetic lasso [MB95]
– edge-aware brush [CPD07,OH08]
– for multi-touch screens [BWB06]
• User indication– bounding box [RKB04]
– scribbles [BJ01, ADA*04, LSS09, LAA08]
![Page 5: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/5.jpg)
Accurate marking can be tedious
![Page 6: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/6.jpg)
Precise input requires skill…
![Page 7: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/7.jpg)
… and patience
![Page 8: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/8.jpg)
Robust approaches: Related work
• soft selection– AppProp [AP08]
– instant propagation [LJH10]
• interactive segmentation– dynamic, iterative graph cut [SUA12]
![Page 9: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/9.jpg)
Summary of related work
• Accurate methods – require precise input– unstable
• Robust methods– not accurate – marking is not intuitive
![Page 10: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/10.jpg)
Ours: intuitive, robust, fast
Input Output
foreground scribblebackground scribble
foreground background
![Page 11: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/11.jpg)
Formulate as labeling problem
prob
abili
ty
fb
v
pixel grid labeled pixel grid
Labeling
![Page 12: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/12.jpg)
Formulate as labeling problem
prob
abili
ty
fb
v
pixel grid labeled pixel grid
t
![Page 13: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/13.jpg)
Histogram of probabilities using input scribbles
fg bg void0.5 0.25 0.25
0.25 0.5 0.25
0.33 0.33 0.33
![Page 14: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/14.jpg)
Labeling: MAP inference in dense CRF [KK2011]
• approximate
• fast: 0.2 s (50K vars)– reduces to bilateral
filtering
![Page 15: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/15.jpg)
Fast inference assumes Gaussian kernel [KK2011]
• Pair-wise potential (between pixels) – linear sum of Gaussians– Gaussians over Euclidean feature space
• We generalize to arbitrary kernels!
![Page 16: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/16.jpg)
Generalizing MAP inference to arbitrary kernels
Deep in the details (see paper) lurks a Gaussian kernel + Euclidean feature space …
K( , ) = exp(-1/2 ( - )T ∑ -1 ( - ))
feature vectors in Euclidean space
![Page 17: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/17.jpg)
Generalizing MAP inference to arbitrary kernels
K( , ) = exp(-1/2 ( - )T ∑ -1 ( - ))
feature vectors in Euclidean space
what if the input is an arbitrary dissimilarity measure between pixels?
D( , )pixels
Deep in the details (see paper) lurks a Gaussian kernel + Euclidean feature space …
![Page 18: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/18.jpg)
Need an Euclidean embedding!
K( , ) = exp(-1/2 ( - )T ∑ -1 ( - ))
D( , )
embedding
![Page 19: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/19.jpg)
Contributions
• Generalized kernel – approx. mean field inference (fully-connected CRF)
• Application: interactive image binary selection– robust to inaccurate input
![Page 20: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/20.jpg)
Overview
inputEuclideanembeddi
ng
dense CRF +
Inference
output
tt`
t
embedded pixels
t
[KK11]
1. image2. scribbles3. dissimilarity
![Page 21: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/21.jpg)
Approximately-Euclidean pixel embeddingpixel pi
pixel pj
dissimilarity matrix
![Page 22: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/22.jpg)
Approximately-Euclidean pixel embeddingpixel pi
pixel pj
qj
qi
dissimilarity matrix
embedding
![Page 23: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/23.jpg)
Approximately-Euclidean pixel embeddingpixel pi
pixel pj
qj
qit
D( , )≈qi - qj 2
For each pi find qi so that
holds for all pixels pi
embedding
![Page 24: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/24.jpg)
Landmark multidimensional scaling (LMDS) [dST02]
• distance matrix might be huge– 1012 elements for 1 MPix image
• stochastic sampling approach– Nystrom approximation
• Complexity– time: O(N(c+p2) + c3)– space: O(Nc)
N: # pixelsc: # stochastic samplesp: dimensionality of embedding
![Page 25: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/25.jpg)
Thank you
![Page 26: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/26.jpg)
You can’t be serious! What about results?
• Importance of embedding• Role of fully-connected CRF (FC-CRF)• Validation• Comparison with related work• Examples
![Page 27: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/27.jpg)
Embedding allows use of arbitrary dissimilaritiesInput Euclidean distance in RGB [KK11]
Chi-squared distance on local histograms + FC-CCRF
![Page 28: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/28.jpg)
Embedding alone is not sufficiently accurate
Chi-squared distance (local histograms)+ nearest neighbour labeling
Input Euclidean distance in RGB [KK11]
Chi-squared distance on local histograms + FC-CCRF
t
![Page 29: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/29.jpg)
Validation: Accurate output for high input errors
Precise output
Precise input
![Page 30: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/30.jpg)
Color dominant vs texture dominant selectionColor dominant Texture dominant
![Page 31: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/31.jpg)
Qualitative comparison
Ours
[LJH10]
[CLT12]
[FFL10]
![Page 32: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/32.jpg)
Quantitative comparison
Ours
![Page 33: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/33.jpg)
Quantitative comparison
Ours
![Page 34: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/34.jpg)
Summary
• Our selection is robust– relies on relative indication of foreground and background
![Page 35: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/35.jpg)
Conclusion
• Most selection algorithms require precise input– ours is relatively robust
• Genaralising dissimilarities is powerful– in context of pairwise potentials for FC-CRF
• Two distance metrics stood out– RGB distance (colour dominant images)– Chi-squared distance on local histograms (texture dominant images)
![Page 36: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/36.jpg)
Thank you
![Page 37: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/37.jpg)
Ours
[LJH10]
[CLT12]
[FFL10]
![Page 38: Accurate Binary Image Selection From Inaccurate User Input Kartic Subr, Sylvain Paris, Cyril Soler, Jan Kautz University College London, Adobe Research,](https://reader030.fdocuments.in/reader030/viewer/2022032723/56649d005503460f949d2e88/html5/thumbnails/38.jpg)
And with human scribbles