CS688: Web-Scale Image Search Keypoint Localization
Transcript of CS688: Web-Scale Image Search Keypoint Localization
![Page 1: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/1.jpg)
CS688: Web-Scale Image Search
Keypoint Localization
Sung-Eui Yoon
(윤성의)
Course URL:
http://sglab.kaist.ac.kr/~sungeui/IR
![Page 2: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/2.jpg)
2
Homework for Every Class
● Go over the next lecture slides
● Come up with one question on what we have discussed today
● 1 for typical questions (that were answered in the class)
● 2 for questions with thoughts or that surprised me
● Write questions at least 4 times before the mid-term● Multiple questions in one time will be counted as one time
● Common questions are addressed at my draft
● Some of questions will be discussed in the class
● If you want to know the answer of your question, ask me or TA on person
![Page 3: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/3.jpg)
3
Homework for Every Class
● Go over recent papers on image search● High quality papers: Papers published at the
top-tier conf. or close it can be presented; e.g., CVPR, ICCV, ECCV, ACM ICMR, ACM MM, ACM SIGGRAPH
● Recent publication: papers published since 2013
● Find and browse two papers, and submit your summary before every beginning of the Thur. class; submit two summaries
● Online submission is possible
● Think about possible team members
● Too late if you think them later..
![Page 4: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/4.jpg)
4
Computer Vision Field: CVPR, ICCV, ECCV
● Handle various computer vision problems
● Get various machine learning techniques from ICML
![Page 5: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/5.jpg)
5
Example: R-CNN [CVPR 14, oral]
Rich feature hierarchies for accurate object detection and semantic segmentation, Slide is from Mr. Lee
![Page 6: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/6.jpg)
6
Example: Localization Networks
● DenseCap, CVPR 16 (oral)
![Page 7: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/7.jpg)
7
● Use bi-linear interpolation that is differentiable and can be used for back-propagation
![Page 8: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/8.jpg)
8
SIGGRPH
● Focus more on useful applications● Wow factor is important
![Page 9: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/9.jpg)
9
Example: Transfiguring Portraits [SIG. 16]
![Page 10: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/10.jpg)
10
Overall System
Input image &
text
Various feature extractions
(vision tech.)
Search tech.
Image process tech.
![Page 11: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/11.jpg)
11
ACM Multimedia and ACM ICMR
● ICMR (Multimedia retrieval)● A recently created conf. since 2011
● Many papers on image/video search and analysis
● IEEE multimedia● The top-tier conf. in multimedia
● Many different topics related to image/video
![Page 12: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/12.jpg)
12
Example: MindFinder, Finding Images by Sketching
● Sketch-based Image Retrieval via Shape Words. ICMR 2015
● Representation for Sketch-Based 3D Model Retrieval. IEEE Signal Processing Letters, 2014
● Indexing Billions of Images for Sketch-based Retrieval. ACM Multimedia 2013
● Efficient Image Contour Detection using Edge Prior. ICME 2013
● The Scale of Edges, in CVPR 2012
![Page 13: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/13.jpg)
13
Class Objective
● Understand locally invariant features● Key point localization
● Harris detector
![Page 14: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/14.jpg)
14
● Identify similar images given a user-specified image or other types of inputs
Content-Based Image Retrieval (CBIR)
Input
Extract image
descriptors (e.g.,
SIFT)
Web-scale
image database
Output
![Page 15: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/15.jpg)
15
Key Components of Image Search
● Image representations
● Indexing algorithms
● Matching methods
● Classification, Localization, etc.● Apply image search (or nearest neighbor
search)
● Data-driven approach
![Page 16: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/16.jpg)
16
Image Representations
● SIFT, GIST, CNN, etc.● Invariant to different transformations
![Page 17: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/17.jpg)
17
Image Retrieval
● At pre-processing, build a database for efficient retrieval at runtime
query
result
Image
feature
Database
![Page 18: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/18.jpg)
18
Image Retrieval
● At pre-processing, build a database for efficient retrieval at runtime
query
result
Image
feature
Database
Index schemes:
vocabulary trees,
hashing, and
inverted files
![Page 19: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/19.jpg)
19
Image Retrieval: Runtime Procedure
query
result
Image
feature
Database
Query image
![Page 20: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/20.jpg)
20
query
result
Image
feature
Database
Query image
Similar image(s)
Image Retrieval: Runtime Procedure
![Page 21: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/21.jpg)
21
Database
Query image
Post-Processing
Shortlist (e.g., 100 images)
..
..
Re-ranking
(spatial verification)
![Page 22: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/22.jpg)
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Image Retrieval with Spatially Constrained Similarity Measure
[Xiaohui Shen, Zhe Lin, Jon Brandt, Shai Avidan and Ying Wu, CVPR 2012]
![Page 23: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/23.jpg)
23
Classification through Image Search
● Image search
● Find images that have smaller distances to the query
● Classification
● Fine classes that have smaller distances to the query
● Utilize labels
● Classification using image search● Naïve Bayes Nearest Neighbor (NBNN)
● Image classification and Retrieval are ONE
![Page 24: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/24.jpg)
24
![Page 25: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/25.jpg)
25
![Page 26: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/26.jpg)
26
![Page 27: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/27.jpg)
27
![Page 28: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/28.jpg)
28
![Page 29: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/29.jpg)
29
![Page 30: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/30.jpg)
30
![Page 31: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/31.jpg)
31
![Page 32: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/32.jpg)
32
![Page 33: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/33.jpg)
33
![Page 34: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/34.jpg)
34
![Page 35: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/35.jpg)
35
![Page 36: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/36.jpg)
36
![Page 37: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/37.jpg)
37
![Page 38: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/38.jpg)
38
![Page 39: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/39.jpg)
39
This lecture
![Page 40: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/40.jpg)
40
Next lecture
![Page 41: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/41.jpg)
41
![Page 42: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/42.jpg)
42
![Page 43: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/43.jpg)
43
![Page 44: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/44.jpg)
44
![Page 45: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/45.jpg)
45
Two Different Directions
● Classical approaches● Manually designed in image processing and
computer vision fields
● Deep learning approaches● Learned approaches, but are inspired by many
prior (manually crafted) approaches
● In this class● We first talk about the classical approaches,
followed by deep learning approaches
![Page 46: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/46.jpg)
46
![Page 47: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/47.jpg)
47
![Page 48: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/48.jpg)
48
![Page 49: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/49.jpg)
49
![Page 50: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/50.jpg)
50
![Page 51: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/51.jpg)
51
![Page 52: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/52.jpg)
52
![Page 53: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/53.jpg)
53
![Page 54: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/54.jpg)
54
![Page 55: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/55.jpg)
55
![Page 56: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/56.jpg)
56
![Page 57: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/57.jpg)
57
![Page 58: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/58.jpg)
58
![Page 59: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/59.jpg)
59
![Page 60: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/60.jpg)
60
![Page 61: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/61.jpg)
61
![Page 62: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/62.jpg)
62
![Page 63: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/63.jpg)
63
Class Objective were:
● Understand locally invariant features● Key point localization
● Harris detector: manually designed detector automatically learned detector using deep learning
![Page 64: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/64.jpg)
64
Next Time..
● Scale invariant region selection
![Page 65: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/65.jpg)
65
Homework for Every Class
● Go over the next lecture slides
● Come up with one question on what we have discussed today
● Go over recent papers on image search, and submit their summary
![Page 66: CS688: Web-Scale Image Search Keypoint Localization](https://reader031.fdocuments.in/reader031/viewer/2022011903/61d6ae0cf794ed03573f320b/html5/thumbnails/66.jpg)
66
Homework for Every Class
● Go over recent papers on image search● High quality papers: Papers published at the
top-tier conf. or close it can be presented; e.g., CVPR, ICCV, ECCV, MM, SIGGRAPH
● Recent publication : papers published since 2013
● Find and browse two papers, and submit your summary before every beginning of the Thur. class
● Think about possible team members
● Too late if you think them later..