Digital Video Library Network Supervisor: Prof. Michael Lyu Student: Ma Chak Kei, Jacky.
-
Upload
claude-willis -
Category
Documents
-
view
213 -
download
1
Transcript of Digital Video Library Network Supervisor: Prof. Michael Lyu Student: Ma Chak Kei, Jacky.
Digital Video Library NetworkDigital Video Library Network
Supervisor: Prof. Michael Lyu
Student: Ma Chak Kei, Jacky
IntroductionIntroduction
• Overview• System Architecture
– Video Server– Indexing Server– Query Server– Client Applications
• Related Technology
OverviewOverview
• Make large video library to be searchable information resources
• Video– Captures the experience of society– News, TV, Movie…etc
• Search and Discovery– Automated extraction of knowledge from
video– Integration of speech, image, and natural
language understanding for library creation and exploration
Information RetrievalInformation Retrieval
• Given a large collection of multimedia records, find similar/interesting things– Allow fast, approximate queries– Find rules/patterns
• Similarity search– Find pairs of documents that are similar– Find medical cases similar to Smith’s– Find pairs of stocks that move in sync
Application AreasApplication Areas
• Education and training• Consumer and business access to news
and information of interest• Entertainment• Interactive television• Meeting/corporate memory• Video conferences
Diverse TechnologiesDiverse Technologies
• Image Understanding• Scene Understanding• Speech Recognition• Metadata/Entity Extraction• Natural Language Processing• More…
– Database, Network, User Interface...
System ArchitectureSystem Architecture
• Component Based– High Extensibility– High Availability– High Performance
• Workstation or Distributed Systems over Internet
System ArchitectureSystem Architecture
Online Process
Offline ProcessVideo Server Indexing ServerIndexing the Video Contents
Query ServerClient Application
Raw
Vid
eo
User Query
Result Set
Form
al Q
uery
Resu
lt S
et
Request
Vid
eo
Deliver V
ideo
F igure 1: System Overview
System ArchitectureSystem Architecture
VideoServer
VideoServer
VideoServer
IndexingServer
IndexingServer
IndexingServer
QueryServer
QueryServer
QueryServer
Figure 2: DVL Network
Video ServerVideo Server
• Specialized in capturing, storing, and delivery videos
• Dual with different video sources• Features:
– Video Storage– Meta-Media Attributes– Video Delivery
Video StorageVideo Storage
• Store segmented video in digital formats• Video segmentation
– Using low-level visual features– Using multimedia cues
• Semantic segmentation– Using audio, visual, textual signals at
different stages– For Example: use audio feature to separate
speech and commercials; then use text analysis to do story-level segmentation
– Require knowledge on the video source
Meta-Media AttributesMeta-Media Attributes
• For information– related to but not “within” the video– impossible to be extracted from the video
• Five baisc types– Production feature– Media feature– Text description– Intellectual property information– References
Video DeliveryVideo Delivery
• Main concern: – number of current clients– quality of services
• Streaming protocol– reduce the latency for starting the video– exploit the error tolerance nature of video
• QoS– User perspective– Application perspective– Transmission perspective
QoS PerspectivesQoS Perspectives
User Perspective:image size,color depth,
voice quality,steady picture, etc
ApplicationPerspective:
delay,jitter,skew,
error rate
TransmissionPerspective:throughput,
delay,delay variance,
error rate
responsetime
transmissioncost
bandwidth,throughput,burstiness,
compression,transporttechnique
delayjitter
Figure 3: The QoS Venn diagram
QoS Processing ModelQoS Processing Model
Access Map Negotiate
Access Map Negotiate
Access Map Negotiate
Network
User
User PerspectiveLayer
ApplicationPerspective Layer
TransmissionPerspective Layer
Figure 4: A QoS processing model
Indexing ServerIndexing Server
• Specialized in indexing the video for retrieval use
• Features to be indexed– Textual Information– Physical Features– Semantic Features
• Advanced indexing on– Video caption– Company logo– Face recognition
Textual InformationTextual Information
• Includes:– Provided meta-media attributes– Generated script by automatic speech
recognition
• Tradition information retrieval for text documents– Lexical analysis– Removal of stopwords– Stemming– Selection of index terms– Construction of term categorization structures
Speech RecognitionSpeech Recognition
Physical FeaturesPhysical Features
• Low-level objects and associated features
• Features indexed– Color– Texture– Shape– Motion– Spatiotemporal structures
Extract Physical FeaturesExtract Physical Features
• Segment the video into separate shots– Consistent background scene– Extract salient video regions and video
objects
• Index video objects with features mentioned
• Advanced video object extraction in MPEG-4
Semantic FeaturesSemantic Features
• More intuitive and direct then physical features
• Probabilistic graphic model– By Hidden Markov Model (HMM) to investigate
the combination of input features that represent an object
– Identify events, objects, and sites– Using multimedia training data– Limit the lifetime of objects to the shot’s duration– Compute probabilities of
P(car AND road| segment of multimedia data)– Higher level HMM between different objects
(Markov chain Monte Carlo method)
Complexity of FeaturesComplexity of Features
Query ServerQuery Server
• Transform user query to formal queries• Natural language processing• Ranking of results• Different IR Models:
– Boolean Model– Vector Model– Probabilistic Model
• Have knowledge of individual Indexing Servers
• Multimedia Portals!
Client ApplicationsClient Applications
• Basic functionality:– Query– Presentation of Results– Video Playback
• Additional functionality:– Linkage to external database– Manipulation of video
MPEG4MPEG4
• Standard to address multimedia contents– Represent units of aural, visual or
audiovisual content as “media objects”– Natural or synthetic origin– Compose the scene by description of media
objects
• Support QoS in a media-object level• Indexing of media-object become easy
MPEG7MPEG7
• Standard to describe the multimedia content data with some degree of interpretation of the semantics
• Act as the interface for multimedia applications– e.g. Between Video Server and Indexing
Server
ConclusionConclusion
• Challenges– Multilingual Processing– Cognitive Processing– Library Interoperability– Intellectual Property– Security Issues
Thank youThank you