1
EE 6850, F'02, Chang, Columbia U 1
Prof. Shih-Fu Chang
EE 6850, Fall 2002Department of Electrical Engineering
Columbia [email protected], http://www.ee.columbia.edu/~sfchang
Course URL: http://www.ee.columbia.edu/~sfchang/course/vis/
Visual Information Systems
EE 6850, F'02, Chang, Columbia U. 2
Definition
� A computer hardware/software system used for � Acquiring and Storing� Indexing and Searching� Manipulating (editing and quality enhancement)� Distributing� Protectinglarge amount of visual information
� Images, video, animations, and associated multimedia� Examples:
� Web Media Search Engines� Home Media Server� Mobile Multimedia Portals
2
VIS Content Chain and Areas of Interest
content management & exchange
Internet
Broadcast users
(filter, navigation)
Internet users
(streaming, search)
Mobile users (browsing, messaging,transcoding)
broadcast
productionaggregation
EE 6850, F'02, Chang, Columbia U. 4
Why it’s important?
� Adoption of Digital Video� New Content Creation/Presentation Tools
� Editor, slide show, production software
� High-Speed Access Networks� New Services and Devices
� Multimedia Messaging� Multimedia Enabled IA� Media portals
� Standards become available� Digital TV, DVD, MPEG-4, MPEG-7, and IPMP
3
EE 6850, F'02, Chang, Columbia U. 5
Example: PDR (Tivo) Any-Time Paradigm
� Time-shift, local storage: 60 hours � Skipping mode� Instant record, live pause, simultaneous
record/playback� Highlight/trail viewing� Search/retrieval, bookmarking, multi-source
comparison/summarization � Personal profile, multi-user profile� Target services, ads, consumer usage data, � Pay per choice e-Commerce
EE 6850, F'02, Chang, Columbia U. 6
Other Example Systems
� Hermitage Museum Web Site (www.hermitagemuseum.org)
� Media Portal: Columbia WebSeek, Google Image Search
� Video Indexing Engine: � News Video: Virage and ABC� Presentation Video: Fuji Xerox� Search sports video by motion: VideoQ� Video mosaic
� Automatic Video Editing
4
EE 6850, F'02, Chang, Columbia U. 7
Hermitage Museum
� Mixed media search tools: keyword and visual
� Virtual tour (using panoramic and 3D view)� Zoom view gallery: multi-resolution� Watermark protection (invisible)� Balanced use of aesthetic, informational, and
technical components� Issues:
� Acquisition, representation, user interfaces, search tools, copyright protection
EE 6850, F'02, Chang, Columbia U. 8
Web Media Portal
� Google Image Search: claim to be the largest search engine for images on the Web� “jpg” 18M images, “gif” 11.6M images
� WebSEEk: (with J. Smith 1997)� Collect and Categorize Images/Videos on the Web
(600K images, 10K videos)
� Issues� Search keywords, display interfaces, more
advanced search tools
5
EE 6850, F'02, Chang, Columbia U. 9
Image/ compression/
attributeextraction
HSV
PPM
featurevectors
icon/
height,
Format
detection
Format
conversionJPG
VideoVideo
colorconversion
GIF
(re-animation)
width, etc.
motionicon
regions,features, attributes
colorspace
WebSEEk: Extracting Visual Features
EE 6850, F'02, Chang, Columbia U. 10
WebSEEk: Extracting Text Features
� Term Extraction — URL, hyperlink text, tags, file names, directory names, surrounding paragraph, etc.
� URL= http://www.mynet.net:80/animals/domestic-beasts/dog37.jpg
� URL= http://camille.gsfc.nasa.gov/rsd/movies2/Shuttle.gif
� URL=http://www.arch.columbia.edu/DDL/projects/amiens/slide6b.gif
� Demo: WebSEEk
6
EE 6850, F'02, Chang, Columbia U. 11
News Video Indexing Systems
Key frames
Linking transcripts to video
Keywordsearch
(Virage & ABC News)
image
image
image
EE 6850, F'02, Chang, Columbia U. 12
VideoQ:Video Search Engine
� automatic motion analysis, region segmentation and tracking
� spatio-temporal search of multiple video objects
� test on a video collection(3000 video clips)(30,000 video objects)
� Demo
7
EE 6850, F'02, Chang, Columbia U. 13
VideoQ System Overview
video scene cut detectionobject
segmentationindexfeatures
keywords
off-line
videoserver
video archiveviewer
user animated sketch
browsing
queryserver
features
on-line
EE 6850, F'02, Chang, Columbia U. 14
Game 1
Game 2
Commercial
Serve 1, base-line play, 2 strokes
Serve 2 Key Frame
Columbia’s Sports Video Navigator and Streaming
image
8
EE 6850, F'02, Chang, Columbia U. 15
Technical Issues
� Digital video compression� Metadata description and browsing� Video Indexing: shot and scene segmentation� Browsing and visualization interfaces� Content-based multimedia search� Fast Indexing and Relevance Feedback� Copyright protection and authentication� Mobile multimedia access� Evaluation: metrics and events
EE 6850, F'02, Chang, Columbia U. 16
Compression Standards
� MPEG-1� 1.5 Mbps, CD-ROM interactive applications
� MPEG-2� 2-10 Mbps, Digital video, e.g., DTV, DVD
� MPEG-4 � Wide range of bitrates� Low-bit-rate video, e.g., wireless � Natural-synthetic hybrid video� High-quality audio
� H-263(+), H-26L� Video conferencing� Internet video
9
Still region SR1: Creation inform a tion Text annotation
Still region SR2: Text annotation Color structure
Still region SR3: Text annotation Matching hint Color structure
Spatial segment decompos i tion: No overlap, gap
Directional spatial segment relation: left
Content S tructure
Agent object AO1: Label Person
Agent object AO2: Label Person
Event EV1: Label Semantic time Semantic place
Concept C1: Label Property Property
Comradeship
Shake hands
Alex Ana
Object-event relation: hasAccompanierOf
Concept-semantic base rel a tion: hasProperty Of
Content Semantics
Object-event relation: hasAgentOf
Metadata: MPEG-7 Description
Segment-semantic base relation: hasMediaPerceptionOf
Segment-semantic base relation: hasMediaSy mbolOf
Photographer: Seungyup Place: Columbia University Time: 19 September 1998
704x480 pixels True color RGB http://www.alex&ana.jpg
Columbia University, All rights reserved
Creation information: Creation Creator Creation corrdinates Creation location Creation date
Media information: Media profile Media format Media instance
Usage unformation: Rights
Content Management
EE 6850, F'02, Chang, Columbia U. 18
Shot boundary detection
� Shot: a continuous take of video content� Task: how to break long video into individual
shot units?� Shot boundary detection based on color,
motion, edge� Demo: sports, sitcom, consumer, news� issues:
� flash, lighting changes, motion, special effects� alignment of captions with shot boundaries
10
EE 6850, F'02, Chang, Columbia U. 19
Challenge: How to find scene boundaries?
transient2:20 4:045:22
Sceneboundary
image image image image image
EE 6850, F'02, Chang, Columbia U. 20
Watermarking and Content Protection
Tx Rx
Watermark Verify thewatermark
WM
� Stenography vs. watermarking� Watermark
� Visible vs. invisible� Spatial vs. frequency domain� Fragile vs. robust vs. semi-fragile
� Applications: copyright, user ID, content ID, authentication, data hiding
image
11
Multimedia Authentication: Producer Identification and Content Integrity Verification
Editor 1 Transcoder 2Transcoder 1
Editor X Transcoder YTx Rx
Tx Rx
verification verification
verificationverificationverification
verification
Editor 1 Transcoder 2Transcoder 1
Editor X Transcoder YTx Rx
Tx RxEditor 1 Transcoder 2Transcoder 1
Editor X Transcoder YTx Rx
Tx Rx
signature
verification
� Is the received content authentic?
� Who signs the specific content?
� Trustable digital government, business, hospitals, etc.
EE 6850, F'02, Chang, Columbia U. 22
Detecting Forgeries
Manipulated Images
AuthenticationOutput
12
EE 6850, F'02, Chang, Columbia U. 23
Copyright Protection & Copy Control
ScrambleTx Rx
Descramble
VCR
� Ownership Identification, Copy Control have to survive multi-stage transcoding, A/D, D/A conversion
� Prevent unauthorized access� Prevent illegal distribution, e.g., digital cinema
EE 6850, F'02, Chang, Columbia U. 24
Issues
� Content feature extraction� Watermark embedding algorithms� System architecture, combining other
security techniques� General rights management systems� Attack methods� Evaluation metrics
13
Mobile Multimedia Access
� Technical issues� Scalable coding, skimming, bandwidth-restricted
adaptive streaming
Broa dca stCont ent
EE 6850, F'02, Chang, Columbia U. 26
Course format
� Weekly lectures� 3 warm-up homeworks
� (video coding, video indexing, image search)
� 1 final project� team projects� mid-term proposal and brief presentation� final report and presentation� may be extended to “Advanced Projects”
� Test data and tools will be provided
14
EE 6850, F'02, Chang, Columbia U. 27
Example Course Projects (1)
� Experiment-type projects� Components for the video testbed
�Scene change�Video content summarization�Multimedia metadata visualization tools
� Speech recognition and text parsing tools� MPEG-7 authoring and parsing tools� Mobile streaming and transcoding� Automatic Content-based video editing
(e.g., muvee.com)
EE 6850, F'02, Chang, Columbia U. 28
Project topics (2)� video indexing
� Echo video visualization combined with 3D graphics� Segmentation of video objects and content based search� Mosaic-based video browsing
� Example: dartfish http://www.dartfish.com/en/solutions/in_showcase.jsp
� Evaluation and comparison� Watermarking: protection and attacks� Similarity metrics
� Survey � content management systems in medical, biological, graphic, music
domains� Music information retrieval� DVD copyright protection standards� MPEG-21
Top Related