Lightning talk micro naratives and sensors for real time monitoring, borko vulikic (undp montenegro)
Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic...
-
Upload
tyrone-turner -
Category
Documents
-
view
215 -
download
0
Transcript of Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic...
![Page 1: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/1.jpg)
Multimedia ProjectsOur Experience
Research Projects
NSF Multimedia Laboratory at Florida Atlantic University
(1995-2001)
Director Dr. Borko Furht
![Page 2: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/2.jpg)
ProjectsContent-Based Image Retrieval Using Relevance
Feedback (Oge Marques) IP Simulcast - An Innovative Video and Audio
Broadcasting Technique Over the Internet (Ray Westwater and Jeff Ice)
XYZ Video Encoding Technique (Ray Westwater & Joshua Greenberg)
An Innovative Motion Estimation Algorithm for MPEG Codec (Joshua Greenberg)
A Fast Content-Based Multimedia Retrieval Technique (P. Saksobhavivat)
Interactive Progressive Encoding System (Joe Celi)
![Page 3: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/3.jpg)
Internet Broadcasting or WebcastingBroadcasting multimedia data (audio
and video) over the Internet - from a server (sender) to a large number of clients (receivers)
Applications include: radio and television broadcasting
real-time broadcasting of critical data distance learning videoconferencing database replication electronic software distribution
![Page 4: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/4.jpg)
Broadcast Pyramid Applied in IP Simulcast
SERVER
Client 1
Client 2
Client 3
Client 2
2
Client 4
Client 5
Client 6
Client 9
Client 10
Client 11
Client 12
Client 13
Client 8
Client 14
Client 7
![Page 5: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/5.jpg)
IP Simulcast - An Innovative Technique for Internet Broadcasting
IP Simulcast reduces the server (or sender) overhead by distributing the load to each client (or receiver)
Each receiver becomes a repeater, which rebroadcasts its received content to two child receivers
The needed network bandwidth for the server is significantly reduced
![Page 6: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/6.jpg)
Characteristics of IP Simulcast
It is a radically different model of digital broadcast, referred to as repeater-server model
The server manages and controls the interconnection of repeaters
Each repeater not only plays back the data stream, but also transmits the data to two other repeaters
IP Simulcast provides guaranteed delivery of packets, which is not the case with IP Multicast
![Page 7: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/7.jpg)
Product: AllCastwww.allcast.com
![Page 8: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/8.jpg)
Broadcasting Tree
Once the AllCast Broadcaster is configured, many users can connect to hear/view content. The bandwidth usage is distributed across the participants, as illustrated by the dynamic, self-healing dissemination tree shown in the AllCast main window.
![Page 9: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/9.jpg)
Microsoft Media Player with AllCast Plug-in
•Users can connect to a broadcast using the Microsoft Windows Media Player together with a small, seamlessly integrated AllCast Plug-in. •The plug-in enables the Windows Media Player to participate in peer-to-peer broadcasts.
![Page 10: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/10.jpg)
XYZ - New Video Compression Technique
The XYZ video compression algorithm is based on 3D Discrete Cosine Transform (DCT)
It provides very high compression ratios and excellent video quality
It is very suitable for real-time video compression
![Page 11: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/11.jpg)
Forming Video Cube for XYZ Compression
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
Frames 0 1 2 3 4 5 6 7
8X8X8videocube
![Page 12: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/12.jpg)
Block Diagram of the XYZ Compression
Forward3-DDCT
Quantizer EntropyCoder
QuantizingTables
HuffmanTable
VideoCube
Compressedvideosequence
![Page 13: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/13.jpg)
Key Encoding Equations Both encoder and decoder are symmetrical, which makes
the algorithm suitable for VLSI implementation
Fuv w CuCvCw f x y z
x u y v z w
zyx
( , , ) , ,
cos cos cos
0
7
0
7
0
7
2 1
16
2 1
16
2 1
16
(4.1)
FuvwFuvw
Quvwq ,,,,
,,
(4.3)
![Page 14: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/14.jpg)
XYZ Versus MPEGVideo
CompressionTechnique
CompressionRatio
NormalizedRMSError
ExecutionTime [min]
(8 frames, 320x240)
XYZ(QT1)
34.5 0.079 6.45
XYZ(QT2)
57.7 0.097 6.45
XYZ(QT3)
70.8 0.105 6.45
XYZ(QT4)
101.7 0.120 6.45
XYZ(QT5)
128.1 0.130 6.45
MPEGLogarithmic Search andError Correction
11.0 0.080 21.35
MPEGExhaustive Search andError Correction
15.6 0.080 163.0
MPEGLogarithmic Search andNo Error Correction
27.0 0.140 21.35
MPEGExhaustive Search andNo Error Correction
32.9 0.125 163.0
![Page 15: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/15.jpg)
Complexity of Video Compression Techniques
CompressionAlgorithm
EncoderComplexity
DecoderComplexity
TotalComplexity
H.261/H.263 970 200 1,170MPEGNo B Frames
750 100 850
MPEG70% B Frames
1,120 120 1,240
XYZ 240 240 480
![Page 16: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/16.jpg)
XYZ Versus MPEG
Original
MPEGCr=11,NRMSE=0.08
MPEG, Motion Est. onlyCr=27, NRMSE=0.14
XYZCr=45, NRMSE=0.079
![Page 17: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/17.jpg)
Examples of XYZ Compression
Original
XYZ-compressedCr=51
![Page 18: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/18.jpg)
Examples of XYZ Compression
OriginalXYZ-compressedCr=110
![Page 19: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/19.jpg)
Sensitivity of the XYZ Algorithm to Various Video Effects
Movie ClipVideo Effect
CompressionRatio
Normalized RMSEError
Dick TracyTypical Motion
34.5 0.079
Interview with theVampireCamera Break
32.5 0.087
Interview with theVampireCamera Panning
26.1 0.085
Total RecallCamera Panning
17.5 0.049
Total RecallCamera Zoom
23.9 0.042
InterceptorFast Motion
26.0 0.025
![Page 20: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/20.jpg)
Characteristics of XYZ Video CompressionXYZ gives significantly better
compression ratios than MPEG for the same quality of video
For similar compression ratios, XYZ gives much better quality than MPEG
XYZ is faster than MPEG (lower complexity)
XYZ is simple for implementation
![Page 21: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/21.jpg)
Applications of the XYZ
Interactive TV and TV set-top boxesTV phoneVideo broadcasting on the InternetVideo-on-demand applicationsVideoconferencingWireless video
![Page 22: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/22.jpg)
TV Phone Videophone is a box on the top of TV with a small Videophone is a box on the top of TV with a small
camera, modem, and video/audio codec.camera, modem, and video/audio codec.
Videophone Videophone
Conventionaltelephone network
TV set TV set
![Page 23: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/23.jpg)
Design of the TV Phone
VIDEOPHONE
TV
Video/audio capture and compression
Video-in
Video/audio out
Video/audio out
Video/audio in
Modem Telephone lines
User'scontrol panel
Video/audiodecompressionand conversion to TV format
VLSI chipsimplement XYZ algorithm
Camera
![Page 24: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/24.jpg)
A Fast Content-Based Multimedia Retrieval Technique
Two main approaches in indexing and retrieval of images and videos
Keyword-based indexing and retrieval
Content-based indexing and retrieval
![Page 25: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/25.jpg)
Keyword-Based Retrieval and Indexing
Uses keywords or descriptive text, which is stored together with images and videos in the database
Retrieval is performed by matching the query, given in the form of keywords, with the stored keywords
This approach is not satisfactory - the text-based description is incomplete, imprecise, and inconsistent in specifying visual information
![Page 26: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/26.jpg)
New Algorithm for Similarity-Based Retrieval of Images
Images in the database are stored as JPEG-compressed images
The user submits a request for search-by-similarity by presenting the desired image.
The algorithm calculates the DC coefficients of this image and creates the histogram of DC coefficients.
The algorithm compares the DC histogram of the submitted image with the DC histograms of the stored images.
![Page 27: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/27.jpg)
Histogram of DC Coefficients for the Image “Elephant”
![Page 28: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/28.jpg)
Comparison of Histograms of DC Coefficients
![Page 29: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/29.jpg)
Example of Similarity-Based Retrieval Using the DC Histograms
![Page 30: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/30.jpg)
Similarity-Based Retrieval of Compressed Video
Partitioning video into clips - video segmentation
Key frame extractionIndexing and retrieval of key frames
![Page 31: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/31.jpg)
DC Histogram Technique Applied for Video Partitioning
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Frame number
NSD
x 1
00 [%
]
![Page 32: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/32.jpg)
Example of Similarity-Based Retrieval of Key Frames Using DC Histograms
![Page 33: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/33.jpg)
Interactive Progressive Encoding System
Users submit requests for imagery to the image database via a graphical user interface
Upon an initial request, a DCT image (version of the image based on DC coefficients only) is transmitted and reconstructed at the user site.
The user can then isolate specific regions of interests within the image and request additional levels of details.
![Page 34: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/34.jpg)
Band Transmission in Interactive JPEG System Based on Spectral Selection
All blocks of the image
Amplitude of Coefficients Blocks of the selected region of the image
Band-1 Band-2 Band-3 Band-4
DC AC1 AC2 AC3 …… AC6 AC7 ………. AC63
Transmission
![Page 35: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/35.jpg)
Prototype System - IPES and Experimental Results
Original image “Airport”
![Page 36: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/36.jpg)
Interactive Progressive Transmission in Four Scans
![Page 37: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/37.jpg)
Selection of Two Regions
![Page 38: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/38.jpg)
Cumulative Number of Transmitted Bits
0
200
400
600
800
1000
1200
1400
1600
1800
2000
1 2 3 4
Scans
Cum
ulat
ive
tran
smitt
ed b
its [K
bits
]
Whole Image
1 Region
2 Regions
![Page 39: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/39.jpg)
Extracted Images From a Group of Images
![Page 40: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/40.jpg)
Applications
Retrieval and transmission of complex images over low bandwidth communication channels (image transmission over the Internet, real-time transmission of medical images)
Archiving and browsing visually lossless image databases (medical imaginary, space exploration and military applications)
![Page 41: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/41.jpg)
Content-Based Retrieval
Large, complex, and ever growing, distributed, mostly unstructured, multimedia repositories
Three ways of retrieving multimedia information:– Free browsing (inefficient, time-consuming, doesn’t
scale well)– Text-based retrieval (relies on metadata, time-
consuming, subjective)– Content-based retrieval (requires intelligent
interpretation of the contents)
![Page 42: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/42.jpg)
Design of MUSE System
Image Analysis
Image Feature Extraction-Color-Shape-Texture
Image Representation & Feature Organization
ImageArchive
User
GUI-Image selection-Result viewing
Probability recalculation & candidate ranking
Feature ExtractionSimilarity comparison
Interactive learning& Display update
Off-line Online
![Page 43: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/43.jpg)
Query By Example
Example image
Best result
Similarity Score [0,1]
![Page 44: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/44.jpg)
Relevance Feedback
Good
Bad
Neither
![Page 45: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/45.jpg)
Relevance Feedback - Next
![Page 46: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/46.jpg)
Technology Behind the MUSE System Feature extraction
Extraction of relevant image features impacts the overall performance of the system.
MUSE uses:– color-related features (color histograms, color
space partitioning and/or quantization, color moments, color coherence vectors)
– texture-related features (Multiresolution Simultaneous Autoregressive Model - MSAR)
– frequency-related features (DFT, DCT)
![Page 47: Multimedia Projects Our Experience Research Projects NSF Multimedia Laboratory at Florida Atlantic University (1995-2001) Director Dr. Borko Furht.](https://reader035.fdocuments.in/reader035/viewer/2022062423/56649ebc5503460f94bc5654/html5/thumbnails/47.jpg)
Technology Behind the MUSE System Bayesian formulation
MUSE is based on a Bayesian framework for relevance feedback.
During each iteration of a MUSE session, the system displays a subset of images from its database, and the user takes an action in response, which the system observes.
Based on the user’s actions, the probability distribution over possible targets is refined. (Most systems refine the user’s query)
The best candidates are then displayed back.