2023! ITEA CANDELA Content Analysis and Networked DELivery Architectures International Review...

2023!

ITEA ITEA CANDELACANDELA

Content Analysis and Networked DELivery ArchitecturesContent Analysis and Networked DELivery Architectures

International ReviewInternational ReviewHelsinki, FinlandHelsinki, Finland

June 16, 2005June 16, 2005

Home Multimedia DemonstrationsHome Multimedia Demonstrations

CANDELA 2Project Presentation CANDELA

......................................................OutlineOutline

• Introduction & PositioningIntroduction & Positioning Video / Multimedia Content Analysis (VCA / MCA) and Video / Multimedia Content Analysis (VCA / MCA) and Networks for Home MultimediaNetworks for Home Multimedia

• WP1: VCA / MCA for Home Multimedia WP1: VCA / MCA for Home Multimedia • Service Unit AV segmentationService Unit AV segmentation• Service Unit Face detectionService Unit Face detection• Service Unit Sport sequence analysisService Unit Sport sequence analysis

• WP2: System ArchitectureWP2: System Architecture• Distributed content analysis (encapsulated Distributed content analysis (encapsulated

components – Service Units), system architecture components – Service Units), system architecture and system solutions for prototyping in-home and system solutions for prototyping in-home networks networks

• WP2: AV content and meta database WP2: AV content and meta database • Database solutions for storing and retrieving featuresDatabase solutions for storing and retrieving features

• Integration with other CANDELA Integration with other CANDELA application application domainsdomains

• Home Multimedia meets Mobile MultimediaHome Multimedia meets Mobile Multimedia

Acquisition

CompressionContentAnalysis

Search & Retrieve

Deliver

Present

Contentdatabase

Metadatadatabase

MPEG7MPEG2/4


......................................................WP1WP1 WP2WP2 WP3WP3

Networkedinfrastructure

Interoperability of media and devices

Demonstrators

Surveillance

Home Multimedia

Medical systems

App

licat

ions

CANDELA introduction & positioningCANDELA introduction & positioningFocus of the Home Multimedia application demonstratorFocus of the Home Multimedia application demonstrator

Philips Research, Philips Research, TU/eTU/e

CANDELA 4Project Presentation CANDELAFinal Review, June 16, 2005, Helsinki, FI

......................................................

WWW

recordrecord

play

CableEPG

Content management

Aim of CANDELA Home Multimedia Aim of CANDELA Home Multimedia Content Management in Consumer Electronics (CE) NetworksContent Management in Consumer Electronics (CE) Networks

The Future of Consumer Electronics In-Home NetworksThe Future of Consumer Electronics In-Home Networks– storage capacity of >1 Tbyte storage capacity of >1 Tbyte

consumer desire for ease-of-use consumer desire for ease-of-use– Distributed processing GigaFlopsDistributed processing GigaFlops

Problem: retrieving and managing AV content Problem: retrieving and managing AV content Required: Required:

Metadata (content descriptors) & their generation (VCA / MCA)Metadata (content descriptors) & their generation (VCA / MCA)System solutions for distributed content analysis (DCA) including media System solutions for distributed content analysis (DCA) including media database solutions and metadata standards for interoperabilitydatabase solutions and metadata standards for interoperability


......................................................

Visualizationwall

User Interface

Cable

WWW

AV

Content & descriptors

Content Management

HDD

Database

MPEG7Storage

VCA for ‘Smart’ content managementVCA for ‘Smart’ content managementReal-time Content Analysis dissemination

Demo at ICME 2005 (Amsterdam, The Netherlands)

Collaboration projects CANDELA / MultimediaN


......................................................WP1: VCA / MCA for home multimedia WP1: VCA / MCA for home multimedia Service Unit (SU) AV Segmentation & Face DetectionService Unit (SU) AV Segmentation & Face Detection

• AV SegmentationAV Segmentation– Auto AV chaptering for CE devicesAuto AV chaptering for CE devices– Annotation tool for Home Videos Annotation tool for Home Videos

• SU Cut Detection (CD) + SU Audio SU Cut Detection (CD) + SU Audio Segmentation (AS) Segmentation (AS)

... results in SU AV Segmentation (AVS)... results in SU AV Segmentation (AVS)

SU CD SU AS

SU ScBD

Media Database

MPEG-7 conversion

User e.g. SOLID / MICROGENESIS

Content from e.g. MICROGENESIS / Bosch

SU FD

• Face DetectionFace Detection– Still & video content managementStill & video content management– Surveillance & Automotive & Basis for Face Surveillance & Automotive & Basis for Face

Recognition: SU Face Detection (FD)Recognition: SU Face Detection (FD)• Face Detector on a commercial Philips productsFace Detector on a commercial Philips products• Benchmarked of various Viola-Jones-based Face Benchmarked of various Viola-Jones-based Face

DetectorsDetectors

CANDELA: content from Co. Microgenesis / SOLID CANDELA: content from Co. Microgenesis / SOLID MPEG-7 output to SOLID / BOSCH / MicrogenesisMPEG-7 output to SOLID / BOSCH / Microgenesis


......................................................

WP1: VCA / MCA for home multimedia WP1: VCA / MCA for home multimedia Service Unit Automatic Sports AV AnalysisService Unit Automatic Sports AV Analysis (Video Coding and Architectures Group / TU Eindhoven)(Video Coding and Architectures Group / TU Eindhoven)

Court detection and camera calibration

Player detectionand tracking

Event analysis

Video input Playing scene detection

1. Get local area2. White pixel rate3. Threshold difference

1. Detect white pixel2. Detect line3. Camera model

1. Build background2. Produce binary map3. Extract players4. Track players

1. Real world speed2. Real world position3. Speed change

Output


......................................................DemoDemo


......................................................OutlineOutline

• Introduction & PositioningIntroduction & Positioning AudioVisual (AV) Content Analysis (CA) and Network for AudioVisual (AV) Content Analysis (CA) and Network for Home MultimediaHome Multimedia

• WP1: AV CA for Home Multimedia WP1: AV CA for Home Multimedia • Service Unit AV content segmentationService Unit AV content segmentation

• Service Unit Face detectionService Unit Face detection

• Service Unit Sport sequence analysisService Unit Sport sequence analysis

• WP2: System ArchitectureWP2: System Architecture• Distributed content analysis (encapsulated components – Distributed content analysis (encapsulated components –

Service Units), system architecture and system solutions Service Units), system architecture and system solutions for prototyping in-home networks for prototyping in-home networks

• WP2: AV content and meta database WP2: AV content and meta database • Database solutions for storing and retrieving featuresDatabase solutions for storing and retrieving features

• Integration with other CANDELA Integration with other CANDELA application domainsapplication domains• Home Multimedia meets Mobile MultimediaHome Multimedia meets Mobile Multimedia

Acquisition


Search & Retrieve

Deliver

Present

Contentdatabase

Metadatadatabase

MPEG7MPEG2/4


......................................................WP2: networked deliveryWP2: networked delivery

• System architecture for the in-home systemSystem architecture for the in-home system– database, terminals, processing componentsdatabase, terminals, processing components– interconnectinterconnect– protocolsprotocols

• The connected devices form the The connected devices form the physical platformphysical platform– content sharingcontent sharing– shared processingshared processing– shared resourcesshared resources

• Software architectureSoftware architecture– service unitsservice units encapsulate functionality encapsulate functionality– this functionality is exposed on the networkthis functionality is exposed on the network– applications applications as compositions of service units (e.g. VCA as compositions of service units (e.g. VCA

applications)applications)


......................................................Example: commercial block detectorExample: commercial block detector

Input

Required metadata: -commercial block detection

AVDecoderSU

Meta YUV SU

AFXSU

MetaSyncSU

Low-level features

Silence Detector SU

CutDetector SU

AV Segmentor SUMid-level

features

Commercial BlockDetector SU

High-level features

MDB

GUI

metadataaudio video

PC2

PC4PC3

Network

PC5

PC6

metadata

video

audio

video

metadatametadata

metadata


......................................................Focus: reliabilityFocus: reliability

• Applications rely on the cooperation of several devices Applications rely on the cooperation of several devices – devices may disappeardevices may disappear– more points of failures more points of failures – integration of 3integration of 3rdrd party VCA/MCA components party VCA/MCA components

• Considered errorsConsidered errors– Service UnitsService Units

• Logical (catchable by a Service Unit itself)Logical (catchable by a Service Unit itself)• Physical (failstop of process running the service)Physical (failstop of process running the service)

– Host (device) breakdownHost (device) breakdown– Communication (network) failureCommunication (network) failure


......................................................ExampleExample

Error detection and recoveryError detection and recovery

1.1. UUnplug a network cable from the systemnplug a network cable from the system

2.2. The health keeperThe health keeper detects the malfunctiondetects the malfunction

3.3. Recovery: tRecovery: the missing component is started he missing component is started on on another nodeanother node and and reconnected reconnected after a timeout has exceededafter a timeout has exceeded

SU1

HM1

PC1PC1 SU2

HM2

PC2PC2

PC3PC3

Health KeeperSU1_new

PC4PC4


......................................................OutlineOutline

• Introduction & PositioningIntroduction & Positioning AudioVisual (AV) Content Analysis (CA) and Network for AudioVisual (AV) Content Analysis (CA) and Network for Home MultimediaHome Multimedia

• WP1: AV CA for Home Multimedia WP1: AV CA for Home Multimedia • Service Unit AV content segmentationService Unit AV content segmentation

• Service Unit Face detectionService Unit Face detection

• Service Unit Sport sequence analysisService Unit Sport sequence analysis

• WP2: System ArchitectureWP2: System Architecture• Distributed content analysis, system architecture and Distributed content analysis, system architecture and

system solutions for prototyping in-home networks system solutions for prototyping in-home networks

• WP2: AV content and meta data databaseWP2: AV content and meta data database• Database solutions for storing and retrieving featuresDatabase solutions for storing and retrieving features

• Integration with other CANDELA Integration with other CANDELA application domainsapplication domains• Home Multimedia meets Mobile MultimediaHome Multimedia meets Mobile Multimedia

Acquisition


Search & Retrieve

Deliver

Present

Contentdatabase

Metadatadatabase

MPEG7MPEG2/4


......................................................WP2: AV content and meta databaseWP2: AV content and meta database

• Allows persistent storage of featuresAllows persistent storage of features• Allows nonlinear features browsing and searchingAllows nonlinear features browsing and searching• Allows non real-time features processingAllows non real-time features processing• Enables features distribution using metadata standardsEnables features distribution using metadata standards• Enables new applicationsEnables new applications• Serves as a reference to evaluate new algorithmsServes as a reference to evaluate new algorithms


......................................................Media Database (MDB) FrameworkMedia Database (MDB) Framework

• Persistent storage and management of:Persistent storage and management of:– digital asset and asset collectionsdigital asset and asset collections– AV featuresAV features– info about people and devicesinfo about people and devices

• Data representation:Data representation:– multiple formats (DIDL, MPEG7, ID3)multiple formats (DIDL, MPEG7, ID3)– dynamic views (annotations)dynamic views (annotations)– views on remote datasets (snapshots)views on remote datasets (snapshots)

• Adaptable :Adaptable :– support for ‘unknown’ metadatasupport for ‘unknown’ metadata– extensible (API, model)extensible (API, model)

• Interfaces:Interfaces:– relationalrelational– object-orientedobject-oriented

Mobile Stationary

wrappers

APIs

objects/ schemas

Asset Mng

PersonMng

AV Features

EPG Mng

DBMSes

databases

MySQL Solid SQLite XML B-tree

local DB snapshots

ODBC SQL libdbi wrappers

APIs

objects/ schemas

Asset Mng

PersonMng

AV Features

EPG Mng

DBMSes

databases


local DB snapshots

ODBC SQL libdbi wrappers

APIs

objects/ schemas

Asset Mng

PersonMng

AV Features

EPG Mng

DBMSes

databases


local DB snapshots

ODBC SQL libdbi


......................................................

Analysis

Content

Metadata

A/V Signal (Solid, Microgenesis)

MPEG7

Integration with other CANDELA demonstratorsIntegration with other CANDELA demonstratorsHome Multimedia meets Mobile MultimediaHome Multimedia meets Mobile Multimedia

e.g. face detectione.g. face detection

Mobile multimedia demo

Home multimedia demo

DB2.5G3G

WWW

WLAN


......................................................Summarizing: resultsSummarizing: results

• VCA algorithms for generating metadataVCA algorithms for generating metadata– video annotation with e.g. face recognition, shot detection, video annotation with e.g. face recognition, shot detection,

tennis scene analysistennis scene analysis

• Different semantic levels: low, mid and high-level features Different semantic levels: low, mid and high-level features – mid level: semantic AV segmentation, face detectionmid level: semantic AV segmentation, face detection– mid & high level: tennis player position & behaviormid & high level: tennis player position & behavior

• Data management system for storing and retrieving AV featuresData management system for storing and retrieving AV features– adaptable for heterogeneous devicesadaptable for heterogeneous devices– interoperable with multiple data formatsinteroperable with multiple data formats

• System architecture System architecture – flexible and reliable composition of components into applicationsflexible and reliable composition of components into applications– distributed processingdistributed processing

2023! ITEA CANDELA Content Analysis and Networked DELivery Architectures International Review...

Documents

Transcript of 2023! ITEA CANDELA Content Analysis and Networked DELivery Architectures International Review...