WbWeb-SlI Dtb dScale Image Databases and Their A pplications

W b S l I D t b dWeb-Scale Image Databases and Their Applicationspp

Sung-Eui YoonKAIST

http://sglab.kaist.ac.kr

Web-Scale Visual Data and Novel ApplicationsApplications

Visual data are widely used for various● Visual data are widely used for various communication and, and are more widely consumed at Web and mobile devicesconsumed at Web and mobile devices● YouTube, Facebook, Flickr, etc.

● Processing them requires scalableProcessing them requires scalable algorithms

●Web-scale visual data can enable new applications

● Examples● Photo tourism● Scene completion● Image-retrieval based image watermarking● Interactive content-aware zooming

OutlineOutlineImage Retrieval based Image● Image Retrieval based Image Watermarking for Large-Scale Image DatabasesDatabases

● Scene Completion using Millions of Photographsg p

● Interactive Content-Aware Zooming● Photo Tourism● Photo Tourism● Conclusions

Image RetrievalImage RetrievalAt pre processing build an database for● At pre-processing, build an database for efficient retrieval at runtime

query

DatabaseDatabase

resultImage

5

gfeature

Image RetrievalImage RetrievalAt pre processing build an database for● At pre-processing, build an database for efficient retrieval at runtime

query

Database

kd-trees or vocabulary Database ocabu a y

trees

resultImage

6

gfeature

Image Retrieval: Runtime ProcedureProcedure

Q iQuery image

query

DatabaseDatabase (kd-tree)

resultImage

7

gfeature

Image Retrieval: Runtime Procedure

Q i

Procedure

Query image

query

DatabaseDatabase (kd-tree)

resultImage

8

gfeature Similar image(s)

Issues of Image-Retrieval for Web Scale Image DatabasesWeb-Scale Image Databases

Accuracy issues● Accuracy issues●Memory issues

Th t t f th t t h i h dl● The state-of-the-art techniques can handle about 10M images in a commodity hardware

●Handling dynamic databases of images●Handling dynamic databases of images● Not much work on efficient handling data

databasesdatabases● Copyright violations of images

● IRIW: Image Retrieval based Image● IRIW: Image Retrieval based Image Watermarking for Large-Scale Image Databases, JongYun Jun, et al., KAIST Tech. R

9

Report

IntroductionIntroduction

<= 2009 Google

10

WatermarkingWatermarkingA th t b d d t ll d t k● A process that embeds data, called watermark● Watermark is integrated into the content itself

● Requires no additional file header

● Resist on conversion of data format● Resist on conversion of data format

11

MotivationProblem

Motivation● Problem

● How to find unauthorized image usages?

12

Possible Approach● Exhaustive watermark matching

Possible Approach● Exhaustive watermark matching

● Sequential one-to-one comparison● Time-consuming job● Time consuming job

WM similarity99% detect25%25%70% fail15% failQuer 15% failQuer

y

13

TerminologyTerminology

hdIsetresult :

settruth ground:RI

positive false : FPnegative false : FN

)(of#Precision RI

)(of#Recall RI

)(of#Precision

R )(of#Recall

I

14

GoalGoalIdentify modified watermarked images in● Identify modified watermarked images in efficient and accurate manner by combining with image retrieval in large-combining with image retrieval in largescale database.

●Main assumption● Dissimilar images have less relevance g

15

Related WorkRelated WorkImage Retrieval● Image Retrieval● D. Lowe. Distinctive image features from scale-

invariant keypoints Computer Vision 2004invariant keypoints. Computer Vision 2004.● D. Nister and H. Stewenius. Scalable recognition

with a vocabulary tree. CVPR 2006.y

● Image Retrieval with Watermarking● Image Retrieval with Watermarking● Lu et al. Image retrieval based on a multipurpose

watermarking scheme. KBIIES 2005.● Xu et al. A new scheme of image retrieval based

upon digital watermarking. ISCSCT 2008

16

Our Approach● Exhaustive watermark matching

Our Approach● Exhaustive watermark matching

● Sequential one-to-one comparison● Time-consuming job● Time consuming job

● Image Retrieval based Image watermarking (IRIW)g g ( )● Reduce search domain by image search● Achieve performance enhancement

Large-scale Image Watermark Large-scale Watermark gDatabase

gretrieval comparison

gDatabase comparison

17

OverviewOverview

Phase 1:Query image IR output list

Phase 1:Image

retrieval

Top-1

Top-2

.

..

Top-1

Phase 2:Watermark extraction

Phase 3:Re-ranking

p

Top-2

.

.. extractiong..

Final output list

18

Phase 1 Image RetrievalPhase 1 – Image RetrievalMain assumption●Main assumption● Dissimilar images have less relevance

● Performance speed-upC t i il i d ll t th● Compute similar images and cull out others

● Accuracy● Detect severely attacked images even though

t k i d (f l ti )watermark is removed (false negative)● Cull out dissimilar images (false positive)

19

Phase 2 Watermark ExtractionPhase 2 – Watermark ExtractionExtract watermarks only from image● Extract watermarks only from image retrieval list and compare the similarity

● Sort output list based on watermark and image similarityimage similarity

9 9 18

9+8

7

8

7

16

False Positive

8

IR t t li t WM t ti O li

21

21 3

FalseNegative

20

IR output list(Image similarity)

WM extraction (Watermark similarity)

Output list

Phase 3 Re rankingHigh ranked images

Phase 3 – Re-ranking●High ranked images

● Have high image similarity● Have high watermark similarity● Have high watermark similarity

B tili ing high anked images e ank● By utilizing high ranked images, re-rank output list based on image similarity

21

ResultResultRuntime performance (10K images)●Runtime performance (10K images)● Exhaustive search : 19 min

● Our approach : Average 5.9 sec●SIFT extraction : 0 34 sec●SIFT extraction : 0.34 sec●Image retrieval : 0.71 sec●WM comparison (30 images) : 4.9 secp ( g )

● 200x performance enhancement● 200x performance enhancement

22

ResultResultAccuracy● Accuracy

● Crop● Crop● Scale● Shear● Shear● Rotate● Noise● Median● JPEG

23

ResultResultAcc ac (100 tests)● Accuracy (100 tests)

set truth ground:I)(of#Precision RI

setresult :R)(of#Precision

R)(of#Recall RI

24

)(of#Recall

I

ConclusionConclusionImage retrieval based image watermarking● Image retrieval based image watermarking● Cull out irrelevant images in terms of image

similaritysimilarity● Can be used with other watermark algorithms

● Two order of magnitude speed-up

●Higher accuracy (small number of FP & FN) ● Cull out irrelevant images (FP)● Cull out irrelevant images (FP)● Detect severely attacked images (FN)● Re-ranking phase (FP & FN)

25

● Re ranking phase (FP & FN)




26




27




28




29

ConclusionsConclusionsVisual data are more widely used for● Visual data are more widely used for various communication and are thus associated at Webassociated at Web

● Processing them requires scalable algorithmsg

●Web-scale visual data can enable new applications

● Examples● Photo tourism● Scene completion● Image-retrieval based image watermarking

30

● Interactive content-aware zooming

WbWeb-SlI Dtb dScale Image Databases and Their A pplications

Documents

Transcript of WbWeb-SlI Dtb dScale Image Databases and Their A pplications