Contextual Image Search
30
Contextual Image Search Wenhao Lu Wenhao Lu , Jingdong Wang , Xian-Sheng Hua, Shengjin Wang , Shipeng Li , Jingdong Wang , Xian-Sheng Hua, Shengjin Wang , Shipeng Li Tsinghua University, Beijing, P. R. China, Tsinghua University, Beijing, P. R. China, Microsoft Research Asia, Beijing, P. R. China, Microsoft Research Asia, Beijing, P. R. China, MM 2011
description
Contextual Image Search. Wenhao Lu , Jingdong Wang , Xian- Sheng Hua , Shengjin Wang , Shipeng Li Tsinghua University, Beijing, P. R. China, Microsoft Research Asia, Beijing, P. R. China,. MM 2011. Outline. System overview Database construction - PowerPoint PPT Presentation
Transcript of Contextual Image Search
Contextual Image SearchContextual Image Search
Wenhao LuWenhao Lu , Jingdong Wang , Xian-Sheng Hua, Shengjin Wang , Shipeng Li , Jingdong Wang , Xian-Sheng Hua, Shengjin Wang , Shipeng Li
Tsinghua University, Beijing, P. R. China, Tsinghua University, Beijing, P. R. China,
Microsoft Research Asia, Beijing, P. R. China, Microsoft Research Asia, Beijing, P. R. China,
Wenhao LuWenhao Lu , Jingdong Wang , Xian-Sheng Hua, Shengjin Wang , Shipeng Li , Jingdong Wang , Xian-Sheng Hua, Shengjin Wang , Shipeng Li
Tsinghua University, Beijing, P. R. China, Tsinghua University, Beijing, P. R. China,
Microsoft Research Asia, Beijing, P. R. China, Microsoft Research Asia, Beijing, P. R. China,
MM 2011
MM 2011
Outline
System overview Database construction Contextual image search with text/image input Experiment Future Work
2
MM 2011
System overview
3
Text input
MM 20114
Image input
System overview
MM 20115
Database construction
MM 20116
Database construction
1. Feature extraction (MSER)
extracts stable regions from the image by considering the change in area w.r.t the change in intensity of a connected component defined
MM 20117
Database construction
2. SIFT descriptor
MM 20118
Database construction
2. SIFT descriptor
MM 20119
Contextual Image Search WithText Input
1. Context Capturing
visual contexts: vision-based page segmentation algorithm (VIPS)
textual contexts: page title / document title local context
MM 201110
vision-based page segmentation
Traditional DOM tree
MM 201111
vision-based page segmentation
VIPS
MM 201112
vision-based page segmentation
Tag cue: <HR>Color cue: background colorText cueSize cue
DOM tree +Visual Info
13
Contextual Image Search WithText Input
2. Contextual Query Augmentation
Goal: remove possible ambiguities Augmented query = query + textual context
Candidate augmented query
evaluate the relevance betweenthe context and augmented query (Okapi BM25)MM 2011
14
MM 2011
2. Contextual Query Augmentation
: extended context (using synonyms, stemming, and so on)
k=2.0, b=0.75
Okapi BM25
~
Contextual Image Search WithText Input
2. Contextual Query Augmentation
Rank score =
: static score (ex. the Web page holding this image)
3. Image Search by Text
15
Contextual Image Search WithText Input
MM 2011
Contextual Reranking
textually contextual reranking
visually contextual reranking
, : discarding the augmented query related words
1. Filter out images whose semantic contents may not be relevant to the query. (compute local textual context and query)
16
MM 2011
Contextual Reranking visually contextual reranking
2. Visual word weight:
Find common pattern
3. Compute similarity
:visual contexts
: an image
: histogram vector of i
: histogram vector of k 17
MM 2011
Overall Ranking
= 0.2
= 0.2
=1
18
MM 2011
Contextual Image Search with Image Input
3
1. Search to annotation
discovers the candidate textual queries using the technique “Annotating images by mining search result” (IEEE 2008)
19
MM 2011
Contextual Image Search with Image Input
3
1. Search to annotation
20
MM 2011
Contextual Image Search with Image Input
3
1. Search to annotation
First : find similar image
Second: surrounding texts of the obtained duplicated images are mined to get a list of candidate textual queries
visual features
semantic features
MM 2011
Contextual Image Search with Image Input
1. Search to annotation
22
MM 2011
Contextual Image Search with Image Input
2. Contextual query identification
calculate ~
23
MM 2011
Experiment
24
15,000,000 images and associated web pages
5 users (level 0~level 3)
MM 2011
Experiment
25
0.95
0.65
nDCG curves
MM 2011
Experiment
26
Visual Result for Text Input
MM 2011
Experiment
27
Visual Result for Text Input (Textual Reranking)
MM 2011
Experiment
28
Visual Result for Text Input (Visual Reranking)
MM 2011
Experiment
29
Visual Result for Image Input
textual query “Van gogh”
MM 2011
Future Work
30
1. More general contextual image search, including mobile image search with wider contexts (e.g., position, time, and history)
2. Extend contextual image search to contextual video search by applying the proposed methodology and investigating extra video contexts
![Contextual Based Image Inpainting: Infer, Match and Translate€¦ · Contextual-based Image Inpainting: Infer, Match, and Translate Yuhang Song*1[0000−0003−4990−2964], Chao](https://static.fdocuments.in/doc/165x107/6005fd10d8769943c60a4ddc/contextual-based-image-inpainting-infer-match-and-translate-contextual-based-image.jpg)
