Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf ·...
Transcript of Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf ·...
![Page 1: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/1.jpg)
Multimedia
Content-based Multimedia Retrieval
Course Code 005636 (Fall 2017)
Prof. S. M. Riazul Islam, Dept. of Computer Engineering, Sejong University, Korea
E-mail: [email protected]
![Page 2: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/2.jpg)
Contents
Overview of Content-based multimedia retrieval
Concepts of Content-based image retrieval
Audio Retrieval
Document Image Analysis and Retrieval
System Architecture
![Page 3: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/3.jpg)
Content-based Image Retrieval (CBIR)
Searching for digital images in large databases
What kinds of databases?
What kinds of queries?
What constitutes a match?
How do we make such searches efficient?
![Page 4: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/4.jpg)
Deep blue sky
Orange sunset
![Page 5: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/5.jpg)
CBIR Applications
Art Collections
e.g. Fine Arts Museum of San Francisco
Medical Image Databases
CT, MRI, Ultrasound, The Visible Human
Scientific Databases
Earth Sciences
General Image Collections for Licensing
The World Wide Web
![Page 6: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/6.jpg)
What is a Query?
An image you already have
A rough sketch you draw
S symbolic description of what you want
![Page 7: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/7.jpg)
CBIR System
Offline Processing
![Page 8: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/8.jpg)
Image Features / Distance Measures
Image Database
Query Image
Distance Measure
Retrieved Images
Image Feature
User
Feature Space
Images
![Page 9: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/9.jpg)
Features
Color (histograms, gridded layout, wavelets)
Texture (Laws, Gabor filters, LBP, polarity)An entity consisting of mutually related pixels and group of pixels
Shape (What preprocessing must occur to get shape?)
Objects and their Relationships This is the most powerful, but you have to be able to recognize the
objects!
Artificial texture Natural texture
![Page 10: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/10.jpg)
Research Objective
Image Database
Query Image Retrieved Images
ImagesObject-oriented
Feature Extraction
User
…
Animals
Buildings
Office Buildings
Houses
Transportation
•Boats
•Vehicles
…
boat
Categories
![Page 11: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/11.jpg)
A Taxonomy of Audio
Sound
Music Other?Speech
Classical
Country
Disco Hip Hop
Jazz
RockSports
AnnouncerFemale
Male
Orchestra
String
Quartet
Choir
Piano
?
![Page 12: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/12.jpg)
Acoustic Modeling
Describes the sounds that
make up speech
Lexicon
Describes which
sequences of speech
sounds make up
valid words
Language Model
Describes the likelihood
of various sequences of
words being spoken
Speech Recognition
Speech Recognition Knowledge Sources
![Page 13: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/13.jpg)
Speech Recognition Process
Pronunciation
Lexicon
Signal Processing
Phonetic
Probability
Estimator
(Acoustic
Model)
Decoder
(Language
Model)WordsSpeech
Grammar
![Page 14: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/14.jpg)
Document Image Analysis
Recognize text (OCR)
convert page images to Unicode
machine-printed, handwritten
Analyze page layout geometry
a 2-D problem (unlike speech, text)
good ‘language-free’ algorithms
Capture logical structure
output marked-up text (XML, etc)
exploit non-textual clues
![Page 15: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/15.jpg)
Video/Image OCR Block Diagram
Text Area
Detection
Text Area
Preprocessing
Commercial
OCR
Video or
Image
UTF8 Text
![Page 16: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/16.jpg)
Text Detection
![Page 17: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/17.jpg)
System Architecture
• Combine video, audio and text retrieval scores
Query
Text Image Audio
Text Score Image Score Audio Score
Retrieval
Agents
Final Score
![Page 18: Multimedia - study.riazulislam.comstudy.riazulislam.com/uploads/3/9/8/5/3985970/slides_14.pdf · What kinds of databases? ... Choir Piano? Acoustic Modeling Describes the sounds that](https://reader031.fdocuments.in/reader031/viewer/2022030420/5aa770c37f8b9aee748c0f8a/html5/thumbnails/18.jpg)
Q&A