ITEA CANDELA Content Analysis and Networked DELivery Architectures
-
Upload
giacomo-franklin -
Category
Documents
-
view
33 -
download
2
description
Transcript of ITEA CANDELA Content Analysis and Networked DELivery Architectures
2023!
ITEA ITEA CANDELACANDELA
Content Analysis and Networked DELivery ArchitecturesContent Analysis and Networked DELivery Architectures
International ReviewInternational ReviewHelsinki, FinlandHelsinki, Finland
June 16, 2005June 16, 2005
Home Multimedia DemonstrationsHome Multimedia Demonstrations
CANDELA 2Project Presentation CANDELA
......................................................OutlineOutline
• Introduction & PositioningIntroduction & Positioning Video / Multimedia Content Analysis (VCA / MCA) and Video / Multimedia Content Analysis (VCA / MCA) and Networks for Home MultimediaNetworks for Home Multimedia
• WP1: VCA / MCA for Home Multimedia WP1: VCA / MCA for Home Multimedia • Service Unit AV segmentationService Unit AV segmentation• Service Unit Face detectionService Unit Face detection• Service Unit Sport sequence analysisService Unit Sport sequence analysis
• WP2: System ArchitectureWP2: System Architecture• Distributed content analysis (encapsulated Distributed content analysis (encapsulated
components – Service Units), system architecture components – Service Units), system architecture and system solutions for prototyping in-home and system solutions for prototyping in-home networks networks
• WP2: AV content and meta database WP2: AV content and meta database • Database solutions for storing and retrieving featuresDatabase solutions for storing and retrieving features
• Integration with other CANDELA Integration with other CANDELA application application domainsdomains
• Home Multimedia meets Mobile MultimediaHome Multimedia meets Mobile Multimedia
Acquisition
CompressionContentAnalysis
Search & Retrieve
Deliver
Present
Contentdatabase
Metadatadatabase
MPEG7MPEG2/4
CANDELA 3Project Presentation CANDELA
......................................................WP1WP1 WP2WP2 WP3WP3
Networkedinfrastructure
Interoperability of media and devices
Demonstrators
Surveillance
Home Multimedia
Medical systems
App
licat
ions
CANDELA introduction & positioningCANDELA introduction & positioningFocus of the Home Multimedia application demonstratorFocus of the Home Multimedia application demonstrator
Philips Research, Philips Research, TU/eTU/e
CANDELA 4Project Presentation CANDELAFinal Review, June 16, 2005, Helsinki, FI
......................................................
WWW
recordrecord
play
CableEPG
Content management
Aim of CANDELA Home Multimedia Aim of CANDELA Home Multimedia Content Management in Consumer Electronics (CE) NetworksContent Management in Consumer Electronics (CE) Networks
The Future of Consumer Electronics In-Home NetworksThe Future of Consumer Electronics In-Home Networks– storage capacity of >1 Tbyte storage capacity of >1 Tbyte
consumer desire for ease-of-use consumer desire for ease-of-use– Distributed processing GigaFlopsDistributed processing GigaFlops
Problem: retrieving and managing AV content Problem: retrieving and managing AV content Required: Required:
Metadata (content descriptors) & their generation (VCA / MCA)Metadata (content descriptors) & their generation (VCA / MCA)System solutions for distributed content analysis (DCA) including media System solutions for distributed content analysis (DCA) including media database solutions and metadata standards for interoperabilitydatabase solutions and metadata standards for interoperability
CANDELA 5Project Presentation CANDELAFinal Review, June 16, 2005, Helsinki, FI
......................................................
Visualizationwall
User Interface
Cable
WWW
AV
Content & descriptors
Content Management
HDD
Database
MPEG7Storage
VCA for ‘Smart’ content managementVCA for ‘Smart’ content managementReal-time Content Analysis dissemination
Demo at ICME 2005 (Amsterdam, The Netherlands)
Collaboration projects CANDELA / MultimediaN
CANDELA 6Project Presentation CANDELAFinal Review, June 16, 2005, Helsinki, FI
......................................................WP1: VCA / MCA for home multimedia WP1: VCA / MCA for home multimedia Service Unit (SU) AV Segmentation & Face DetectionService Unit (SU) AV Segmentation & Face Detection
• AV SegmentationAV Segmentation– Auto AV chaptering for CE devicesAuto AV chaptering for CE devices– Annotation tool for Home Videos Annotation tool for Home Videos
• SU Cut Detection (CD) + SU Audio SU Cut Detection (CD) + SU Audio Segmentation (AS) Segmentation (AS)
... results in SU AV Segmentation (AVS)... results in SU AV Segmentation (AVS)
SU CD SU AS
SU ScBD
Media Database
MPEG-7 conversion
User e.g. SOLID / MICROGENESIS
Content from e.g. MICROGENESIS / Bosch
SU FD
• Face DetectionFace Detection– Still & video content managementStill & video content management– Surveillance & Automotive & Basis for Face Surveillance & Automotive & Basis for Face
Recognition: SU Face Detection (FD)Recognition: SU Face Detection (FD)• Face Detector on a commercial Philips productsFace Detector on a commercial Philips products• Benchmarked of various Viola-Jones-based Face Benchmarked of various Viola-Jones-based Face
DetectorsDetectors
CANDELA: content from Co. Microgenesis / SOLID CANDELA: content from Co. Microgenesis / SOLID MPEG-7 output to SOLID / BOSCH / MicrogenesisMPEG-7 output to SOLID / BOSCH / Microgenesis
CANDELA 7Project Presentation CANDELAFinal Review, June 16, 2005, Helsinki, FI
......................................................
WP1: VCA / MCA for home multimedia WP1: VCA / MCA for home multimedia Service Unit Automatic Sports AV AnalysisService Unit Automatic Sports AV Analysis (Video Coding and Architectures Group / TU Eindhoven)(Video Coding and Architectures Group / TU Eindhoven)
Court detection and camera calibration
Player detectionand tracking
Event analysis
Video input Playing scene detection
1. Get local area2. White pixel rate3. Threshold difference
1. Detect white pixel2. Detect line3. Camera model
1. Build background2. Produce binary map3. Extract players4. Track players
1. Real world speed2. Real world position3. Speed change
Output
CANDELA 8Project Presentation CANDELAFinal Review, June 16, 2005, Helsinki, FI
......................................................DemoDemo
CANDELA 9Project Presentation CANDELAFinal Review, June 16, 2005, Helsinki, FI
......................................................OutlineOutline
• Introduction & PositioningIntroduction & Positioning AudioVisual (AV) Content Analysis (CA) and Network for AudioVisual (AV) Content Analysis (CA) and Network for Home MultimediaHome Multimedia
• WP1: AV CA for Home Multimedia WP1: AV CA for Home Multimedia • Service Unit AV content segmentationService Unit AV content segmentation
• Service Unit Face detectionService Unit Face detection
• Service Unit Sport sequence analysisService Unit Sport sequence analysis
• WP2: System ArchitectureWP2: System Architecture• Distributed content analysis (encapsulated components – Distributed content analysis (encapsulated components –
Service Units), system architecture and system solutions Service Units), system architecture and system solutions for prototyping in-home networks for prototyping in-home networks
• WP2: AV content and meta database WP2: AV content and meta database • Database solutions for storing and retrieving featuresDatabase solutions for storing and retrieving features
• Integration with other CANDELA Integration with other CANDELA application domainsapplication domains• Home Multimedia meets Mobile MultimediaHome Multimedia meets Mobile Multimedia
Acquisition
CompressionContentAnalysis
Search & Retrieve
Deliver
Present
Contentdatabase
Metadatadatabase
MPEG7MPEG2/4
CANDELA 10Project Presentation CANDELAFinal Review, June 16, 2005, Helsinki, FI
......................................................WP2: networked deliveryWP2: networked delivery
• System architecture for the in-home systemSystem architecture for the in-home system– database, terminals, processing componentsdatabase, terminals, processing components– interconnectinterconnect– protocolsprotocols
• The connected devices form the The connected devices form the physical platformphysical platform– content sharingcontent sharing– shared processingshared processing– shared resourcesshared resources
• Software architectureSoftware architecture– service unitsservice units encapsulate functionality encapsulate functionality– this functionality is exposed on the networkthis functionality is exposed on the network– applications applications as compositions of service units (e.g. VCA as compositions of service units (e.g. VCA
applications)applications)
CANDELA 11Project Presentation CANDELAFinal Review, June 16, 2005, Helsinki, FI
......................................................Example: commercial block detectorExample: commercial block detector
Input
Required metadata: -commercial block detection
AVDecoderSU
Meta YUV SU
AFXSU
MetaSyncSU
Low-level features
Silence Detector SU
CutDetector SU
AV Segmentor SUMid-level
features
Commercial BlockDetector SU
High-level features
MDB
GUI
metadataaudio video
PC2
PC4PC3
Network
PC5
PC6
metadata
video
audio
video
metadatametadata
metadata
CANDELA 12Project Presentation CANDELAFinal Review, June 16, 2005, Helsinki, FI
......................................................Focus: reliabilityFocus: reliability
• Applications rely on the cooperation of several devices Applications rely on the cooperation of several devices – devices may disappeardevices may disappear– more points of failures more points of failures – integration of 3integration of 3rdrd party VCA/MCA components party VCA/MCA components
• Considered errorsConsidered errors– Service UnitsService Units
• Logical (catchable by a Service Unit itself)Logical (catchable by a Service Unit itself)• Physical (failstop of process running the service)Physical (failstop of process running the service)
– Host (device) breakdownHost (device) breakdown– Communication (network) failureCommunication (network) failure
CANDELA 13Project Presentation CANDELAFinal Review, June 16, 2005, Helsinki, FI
......................................................ExampleExample
Error detection and recoveryError detection and recovery
1.1. UUnplug a network cable from the systemnplug a network cable from the system
2.2. The health keeperThe health keeper detects the malfunctiondetects the malfunction
3.3. Recovery: tRecovery: the missing component is started he missing component is started on on another nodeanother node and and reconnected reconnected after a timeout has exceededafter a timeout has exceeded
SU1
HM1
PC1PC1 SU2
HM2
PC2PC2
PC3PC3
Health KeeperSU1_new
PC4PC4
CANDELA 14Project Presentation CANDELA
......................................................OutlineOutline
• Introduction & PositioningIntroduction & Positioning AudioVisual (AV) Content Analysis (CA) and Network for AudioVisual (AV) Content Analysis (CA) and Network for Home MultimediaHome Multimedia
• WP1: AV CA for Home Multimedia WP1: AV CA for Home Multimedia • Service Unit AV content segmentationService Unit AV content segmentation
• Service Unit Face detectionService Unit Face detection
• Service Unit Sport sequence analysisService Unit Sport sequence analysis
• WP2: System ArchitectureWP2: System Architecture• Distributed content analysis, system architecture and Distributed content analysis, system architecture and
system solutions for prototyping in-home networks system solutions for prototyping in-home networks
• WP2: AV content and meta data databaseWP2: AV content and meta data database• Database solutions for storing and retrieving featuresDatabase solutions for storing and retrieving features
• Integration with other CANDELA Integration with other CANDELA application domainsapplication domains• Home Multimedia meets Mobile MultimediaHome Multimedia meets Mobile Multimedia
Acquisition
CompressionContentAnalysis
Search & Retrieve
Deliver
Present
Contentdatabase
Metadatadatabase
MPEG7MPEG2/4
CANDELA 15Project Presentation CANDELA
......................................................WP2: AV content and meta databaseWP2: AV content and meta database
• Allows persistent storage of featuresAllows persistent storage of features• Allows nonlinear features browsing and searchingAllows nonlinear features browsing and searching• Allows non real-time features processingAllows non real-time features processing• Enables features distribution using metadata standardsEnables features distribution using metadata standards• Enables new applicationsEnables new applications• Serves as a reference to evaluate new algorithmsServes as a reference to evaluate new algorithms
CANDELA 16Project Presentation CANDELA
......................................................Media Database (MDB) FrameworkMedia Database (MDB) Framework
• Persistent storage and management of:Persistent storage and management of:– digital asset and asset collectionsdigital asset and asset collections– AV featuresAV features– info about people and devicesinfo about people and devices
• Data representation:Data representation:– multiple formats (DIDL, MPEG7, ID3)multiple formats (DIDL, MPEG7, ID3)– dynamic views (annotations)dynamic views (annotations)– views on remote datasets (snapshots)views on remote datasets (snapshots)
• Adaptable :Adaptable :– support for ‘unknown’ metadatasupport for ‘unknown’ metadata– extensible (API, model)extensible (API, model)
• Interfaces:Interfaces:– relationalrelational– object-orientedobject-oriented
Mobile Stationary
wrappers
APIs
objects/ schemas
Asset Mng
PersonMng
AV Features
EPG Mng
DBMSes
databases
MySQL Solid SQLite XML B-tree
local DB snapshots
ODBC SQL libdbi wrappers
APIs
objects/ schemas
Asset Mng
PersonMng
AV Features
EPG Mng
DBMSes
databases
MySQL Solid SQLite XML B-tree
local DB snapshots
ODBC SQL libdbi wrappers
APIs
objects/ schemas
Asset Mng
PersonMng
AV Features
EPG Mng
DBMSes
databases
MySQL Solid SQLite XML B-tree
local DB snapshots
ODBC SQL libdbi
CANDELA 17Project Presentation CANDELA
......................................................
Analysis
Content
Metadata
A/V Signal (Solid, Microgenesis)
MPEG7
Integration with other CANDELA demonstratorsIntegration with other CANDELA demonstratorsHome Multimedia meets Mobile MultimediaHome Multimedia meets Mobile Multimedia
e.g. face detectione.g. face detection
Mobile multimedia demo
Home multimedia demo
DB2.5G3G
WWW
WLAN
CANDELA 18Project Presentation CANDELA
......................................................Summarizing: resultsSummarizing: results
• VCA algorithms for generating metadataVCA algorithms for generating metadata– video annotation with e.g. face recognition, shot detection, video annotation with e.g. face recognition, shot detection,
tennis scene analysistennis scene analysis
• Different semantic levels: low, mid and high-level features Different semantic levels: low, mid and high-level features – mid level: semantic AV segmentation, face detectionmid level: semantic AV segmentation, face detection– mid & high level: tennis player position & behaviormid & high level: tennis player position & behavior
• Data management system for storing and retrieving AV featuresData management system for storing and retrieving AV features– adaptable for heterogeneous devicesadaptable for heterogeneous devices– interoperable with multiple data formatsinteroperable with multiple data formats
• System architecture System architecture – flexible and reliable composition of components into applicationsflexible and reliable composition of components into applications– distributed processingdistributed processing