OC ComSig Chapter

Nov. 14, 2001

MPEG-7MPEG-7 ISO 15938 ISO 15938

Multimedia Content Description InterfaceMultimedia Content Description Interface

2

MPEG-7MPEG-7

Dwight BorsesDwight BorsesMTSMTSField Applications EngineeringField Applications EngineeringNational Semiconductor Corp.National Semiconductor Corp.

3

How Much Information?How Much Information?

• The world’s total The world’s total yearlyyearly

production of production of print, film, print, film,

opticaloptical, and , and magneticmagnetic content content

would require roughly 1.5 would require roughly 1.5

billion GB (billion GB (1.5EB1.5EB) of storage.) of storage.

• This is equivalent to This is equivalent to 250MB250MB

per person for every man, per person for every man,

woman, and child on earth.woman, and child on earth.

How Much Information Report

http://www.sims.berkeley.edu/how-much-info

4

Digital InformationDigital Information

• Increasingly, Increasingly, individualsindividuals produce their own produce their own contentcontent

• Of all information produced in the worldOf all information produced in the world– 93%93% is stored in digital form is stored in digital form– HD in stand-alone PCs account for HD in stand-alone PCs account for 55%55% of of

total storage shipped each yeartotal storage shipped each year• Over Over 80 billion photographs80 billion photographs are taken annually are taken annually

– >400 petabytes >400 petabytes – > 80 million times storage required for text> 80 million times storage required for text

*Peta = 1015

5

Information: IndividualsInformation: Individuals

ITEMITEM AMOUNT AMOUNT TERABYTES*TERABYTES*

PhotosPhotos 80 billion images 80 billion images 410,000410,000Home VideoHome Video 1.4 billion tapes 1.4 billion tapes 300,000300,000X-RaysX-Rays 2 billion images 2 billion images 17,200 17,200Hard disksHard disks 200 million installed 200 million installed 13,760 13,760

TOTALTOTAL 740,960740,960

*Tera = 1012

6

Information: PublishedInformation: Published

ITEMITEM AMOUNT AMOUNT TERABYTESTERABYTESBooksBooks 968,735968,735 8 8NewspapersNewspapers 22,643 22,643 25 25JournalsJournals 40,000 40,000 2 2MagazinesMagazines 80,000 80,000 10 10NewslettersNewsletters 40,000 40,000 0.2 0.2Office DocumentsOffice Documents 7.5E9 7.5E9 195195CinemaCinema 4,000 4,000 16 16Music CDsMusic CDs 90,000 90,000 6 6Data CDsData CDs 1,000 1,000 3 3DVD-videoDVD-video 5,000 5,000 22 22

TOTALTOTAL 285285

7

Information: FilmInformation: Film

ITEMITEM UNITS UNITS DIGITALDIGITAL TOTAL TOTAL

CONVERSIONCONVERSION PETABYTES PETABYTES

PhotographyPhotography 82E9 82E9 5Mb/photo5Mb/photo 410410

Motion Pictures 4,000Motion Pictures 4,000 4Gb/movie4Gb/movie 0.0160.016

X-RaysX-Rays 2.16E9 2.16E9 8Mb/radiograph 17.28Mb/radiograph 17.2

ALL FILM TOTALALL FILM TOTAL 427.216 427.216

*Peta = 1015

8

MPEG Family of Standards (1)MPEG Family of Standards (1)

• MPEG-1MPEG-1((19921992): for the ): for the storagestorage and and

retrievalretrieval of moving pictures and audio of moving pictures and audio

on storage media. on storage media. • MPEG-2 MPEG-2 ((19951995): for ): for digital televisiondigital television, the , the

response for the response for the satellitesatellite broadcasting broadcasting

and and cablecable television industries in their television industries in their

transition from analog to digital formats.transition from analog to digital formats.

9

MPEG CompressionMPEG Compression

MPEG encoding produces 3 types of framesMPEG encoding produces 3 types of frames• I-Frame “Intracoded”I-Frame “Intracoded”

– One complete video imageOne complete video image– No other images needed to viewNo other images needed to view– Contains the Contains the most datamost data of any type of any type

• P-Frame “Forward Predicted”P-Frame “Forward Predicted”– Encodes the changes from a previous frameEncodes the changes from a previous frame– Previous frame needed to ‘view’Previous frame needed to ‘view’

• B-Frame “Bidirectionally Predicted”B-Frame “Bidirectionally Predicted”– Encodes changes from a previous or future frameEncodes changes from a previous or future frame– Contains the Contains the least dataleast data (25% of an “I”) (25% of an “I”)– Previous or future frames needed to ‘view’Previous or future frames needed to ‘view’

10

Frame DependencyFrame Dependency

11

MPEG Family of Standards (2)MPEG Family of Standards (2)

MPEG-4 MPEG-4 (1998 v.1, 1999 v.2)(1998 v.1, 1999 v.2)• First real multimedia representation standardFirst real multimedia representation standard• Encodes content as independent objects Encodes content as independent objects • Enables those objects to be manipulated Enables those objects to be manipulated

individually or collectively on an audio visual individually or collectively on an audio visual

scenescene• Allows interactivityAllows interactivity

12

Extension in PurposeExtension in Purpose

• MPEG-1, -2, and -4MPEG-1, -2, and -4– Make content availableMake content available

• MPEG-7MPEG-7– Lets you find the content you needLets you find the content you need

• MPEG-21MPEG-21– Describes “big picture” across wide range of Describes “big picture” across wide range of

networks and devicesnetworks and devices

13

MPEG-3, -5, and –6 MPEG-3, -5, and –6 ??????

• MPEG-3MPEG-3 existed to enable HDTV existed to enable HDTV– Accomplished with tools of MPEG-2Accomplished with tools of MPEG-2– Work item abandonedWork item abandoned

• -1, -2, -4, -1, -2, -4, -5 or -8-5 or -8 ??????– MPEG decided NOT to follow either logical MPEG decided NOT to follow either logical

expansionexpansion– Chose number 7 insteadChose number 7 instead

14

MPEG-21 MPEG-21 ????????

• Comprehensive and flexible framework for the Comprehensive and flexible framework for the 2121stst CenturyCentury– Quality of ServiceQuality of Service– Rights ManagementRights Management– E-CommerceE-Commerce

• Efficient multimedia resource use across networks Efficient multimedia resource use across networks and devicesand devices

• Key concern is processor loading in network Key concern is processor loading in network terminalsterminals

• Draft committee stage expected by Dec 2001Draft committee stage expected by Dec 2001

15

MPEG-7 ISO 15938MPEG-7 ISO 15938

Scope of MPEG-7Scope of MPEG-7

Descriptionconsumption

DescriptionconsumptionDescriptionDescriptionDescription

generation

Descriptiongeneration

Research andfuture competition

A standard for describing features of multimedia content.Completion target: October, 2001 ??????

16

MPEG-7 Will Not …MPEG-7 Will Not …

• Standardize the extraction of Audiovisual Standardize the extraction of Audiovisual descriptions/featuresdescriptions/features

• Specify the software programs that can use Specify the software programs that can use the descriptionsthe descriptions

17

MPEG-7 ISO 15938MPEG-7 ISO 15938

Feature SearchExtraction Engine

MPEG-7Description

standardization

MPEG-7 Scope:Description Schemes (DSs)Descriptors (Ds)Language (DDL)Ref: MPEG-7 Concepts

Search Engine:Searching & filteringClassificationManipulationSummarization Indexing

Feature Extraction:Content analysis (D, DS)Feature extraction (D, DS)Annotation tools (DS)Authoring (DS)

18

Goals and ObjectivesGoals and Objectives

• Provide interoperability among Provide interoperability among systemssystems and and

applicationsapplications used in generation, management, used in generation, management,

distribution and consumption of audio-visual distribution and consumption of audio-visual

content descriptions.content descriptions.• Help Help usersusers or or applicationsapplications to identify, retrieve, to identify, retrieve,

or filter audiovisual information with or filter audiovisual information with

descriptions of descriptions of streamedstreamed or or storedstored media. media.

19

MPEG-7 ContextMPEG-7 Context

• Audiovisual information used to be consumed Audiovisual information used to be consumed directly by human beingsdirectly by human beings

• Increasingly created, exchanged, retrieved, re-Increasingly created, exchanged, retrieved, re-used by computational systemsused by computational systems

• Representations that allow some degree of Representations that allow some degree of interpretation of the information’s meaning can interpretation of the information’s meaning can be accessed and processed by computerbe accessed and processed by computer

20

MPEG-7 Constituent MPEG-7 Constituent ComponentsComponents

• ISO/IEC 15938-1 MPEG-7 ISO/IEC 15938-1 MPEG-7 SystemsSystems• ISO/IEC 15938-2 MPEG-7 ISO/IEC 15938-2 MPEG-7 DDL DDL (Description Definition Language)(Description Definition Language)

• ISO/IEC 15938-3 MPEG-7 ISO/IEC 15938-3 MPEG-7 VisualVisual• ISO/IEC 15938-4 MPEG-7 ISO/IEC 15938-4 MPEG-7 AudioAudio• ISO/IEC 15938-5 MPEG-7 ISO/IEC 15938-5 MPEG-7 MDS MDS (Multimedia Description Schemes)(Multimedia Description Schemes)

• ISO/IEC 15938-6 MPEG-7 ISO/IEC 15938-6 MPEG-7 Reference SoftwareReference Software• ISO/IEC 15938-7 MPEG-7 ISO/IEC 15938-7 MPEG-7 ConformanceConformance

21

Comprehensive AV DescriptionsComprehensive AV Descriptions

• CatalogCatalog– Title, Creator, RightsTitle, Creator, Rights

• SemanticsSemantics– Who, what, when, where of objects and eventsWho, what, when, where of objects and events

• Structural features of AV contentStructural features of AV content– Color of image, timbre of soundColor of image, timbre of sound

• Leverage AV data representations Leverage AV data representations – MPEG-1, -2, -4MPEG-1, -2, -4

22

InteroperabilityInteroperability

• Uses XML Schema for content descriptionUses XML Schema for content description– Over 100 XML industry Standard GroupsOver 100 XML industry Standard Groups– XML Repository at XML Repository at www.xml.orgwww.xml.org

• Groups with similar Objectives to MPEG-7Groups with similar Objectives to MPEG-7– Society of Motion Picture and Television Engineers (SMPTE) Society of Motion Picture and Television Engineers (SMPTE)

[Metadata Dictionary][Metadata Dictionary]– European Broadcasting Union (EBU) European Broadcasting Union (EBU) [P/Meta][P/Meta]– Dublin Dublin [Core][Core]– Digital Imaging Group (DIG)Digital Imaging Group (DIG)– TV-AnytimeTV-Anytime– Ohio Online Computer Center / Research Libraries Group Ohio Online Computer Center / Research Libraries Group

(OCLC/RLG)(OCLC/RLG)Similar approaches with notable divergence from MPEG-7Similar approaches with notable divergence from MPEG-7

23

MPEG-7 Standardized ToolsMPEG-7 Standardized Tools

• Enable detailed structural descriptionEnable detailed structural description– DescriptorsDescriptors– Description schemesDescription schemes– LanguageLanguage

• Different GranularityDifferent Granularity– Region, Image, Video Segment, CollectionRegion, Image, Video Segment, Collection

• Different AreasDifferent Areas– Content description, management, Content description, management,

organization, navigationorganization, navigation

24

MPEG-7 ApplicationsMPEG-7 Applications

• Support and facilitateSupport and facilitate– Media portalsMedia portals– Content broadcastingContent broadcasting– Ubiquitous multimediaUbiquitous multimedia

• Multimedia processing important to end userMultimedia processing important to end user• Multimedia processing important to providers Multimedia processing important to providers

of service and contentof service and content

25

MPEG-7 Data Applications (1)MPEG-7 Data Applications (1)

• Play a few notes on a keyboard and retrieve a list of Play a few notes on a keyboard and retrieve a list of musical pieces similar to the required tune, or images musical pieces similar to the required tune, or images matching the notes in a certain way, e.g. in terms of matching the notes in a certain way, e.g. in terms of emotions.emotions.

• Draw a few lines on a screen and find a set of images Draw a few lines on a screen and find a set of images containing similar graphics, logos, ideograms,...containing similar graphics, logos, ideograms,...

• Define objects, including color patches or textures and Define objects, including color patches or textures and retrieve examples among which you select the retrieve examples among which you select the interesting objects to compose your design.interesting objects to compose your design.

26

MPEG-7 Data Applications (2)MPEG-7 Data Applications (2)

• On a given set of multimedia objects, describe On a given set of multimedia objects, describe movements and relations between objects and so movements and relations between objects and so search for animations fulfilling the described temporal search for animations fulfilling the described temporal and spatial relations.and spatial relations.

• Describe actions and get a list of scenarios containing Describe actions and get a list of scenarios containing such actions.such actions.

• Using an excerpt of Pavarotti’s voice, obtaining a list of Using an excerpt of Pavarotti’s voice, obtaining a list of Pavarotti’s records, video clips where Pavarotti is Pavarotti’s records, video clips where Pavarotti is singing and photographic material portraying Pavarotti.singing and photographic material portraying Pavarotti.

27

Some Application DomainsSome Application Domainswith Applicationswith Applications

• Digital LibrariesDigital Libraries– Image catalog, musical dictionary, biomedical imagingImage catalog, musical dictionary, biomedical imaging

• Multimedia editingMultimedia editing– Media authoring, personal electronic news serviceMedia authoring, personal electronic news service

• Cultural ServicesCultural Services– History museums, art galleriesHistory museums, art galleries

• Multimedia directory servicesMultimedia directory services– Yellow pages, tourist geographical information servicesYellow pages, tourist geographical information services

• Broadcast media selectionBroadcast media selection– Radio channel, TV channelRadio channel, TV channel

28

The Ds of MPEG-7The Ds of MPEG-7

• Audio-Visual Descriptor (Audio-Visual Descriptor (DD))

• Description Schemes (Description Schemes (DSDSs)s)

• Description Definition Language (Description Definition Language (DDLDDL))

29

Relation Between the Different Relation Between the Different MPEG-7 ElementsMPEG-7 Elements

DS

defined in standardnot in standard;defined using DDL

DS

DS DDD

DD

DDL

30

MPEG-7 Terminology: MPEG-7 Terminology: DataData

• Audio-visual Audio-visual information described using information described using

MPEG-7MPEG-7 without regard to storage, coding, without regard to storage, coding,

display, transmission, medium or technologydisplay, transmission, medium or technology

• Intended to be sufficiently broad to encompass Intended to be sufficiently broad to encompass

graphics, still images, video, film, music, graphics, still images, video, film, music,

speech, sounds, text, … speech, sounds, text, …

31

Data ExamplesData Examples

• MPEG-4 streamMPEG-4 stream• Video tapeVideo tape• CD containing musicCD containing music• Sound or speechSound or speech• Picture printed on paperPicture printed on paper• Interactive multimedia installation on the webInteractive multimedia installation on the web

32

MPEG-7 Terminology: MPEG-7 Terminology: FeatureFeature

• Distinctive characteristicDistinctive characteristic of data signifying of data signifying

something to someonesomething to someone

• Cannot be compared without meaningful Cannot be compared without meaningful

feature representation (feature representation (descriptordescriptor) and its ) and its

instantiation (instantiation (descriptor valuedescriptor value))

33

Feature ExamplesFeature Examples

• Color of an imageColor of an image• Pitch of a speech segmentPitch of a speech segment• Rhythm of an audio segmentRhythm of an audio segment• Camera motion in a videoCamera motion in a video• Style of a videoStyle of a video• Title of a movieTitle of a movie• Actors in a movieActors in a movie

34

MPEG-7 Terminology: MPEG-7 Terminology: Descriptor (D)Descriptor (D)

• RepresentationRepresentation of a of a FeatureFeature• Defines syntax and semantics of the Defines syntax and semantics of the FeatureFeature

representationrepresentation• Allows evaluation of corresponding feature by Allows evaluation of corresponding feature by

means of the means of the Descriptor ValueDescriptor Value• Several Several DescriptorsDescriptors may represent a single may represent a single

feature by addressing different relevant feature by addressing different relevant requirementsrequirements

35

Descriptor ExampleDescriptor Example

Color Color FeatureFeature• Color histogramColor histogram• Average of frequency componentsAverage of frequency components• Motion fieldMotion field• Text of the titleText of the title

36

Visual Descriptors

Color Texture MotionShape

• Contour Shape

• Region Shape

• 2D/3D shape

• 3D shape

• Camera motion

• Motion Trajectory

• Parametric motion

• Motion Activity

• Texture Browsing

• Homogeneous texture

• Edge Histogram

1. Histogram

• Scalable Color

• Color Structure

• GOF/GOP

2. Dominant Color

3. Color Layout Face recognition

37

Shape DescriptorsShape Descriptors

• Contour shapeContour shape• Region shapeRegion shape

Contour-basedshape descriptor

Region-basedshape descriptor

38

Motion DescriptorsMotion Descriptors

Video Segment

Camera Motion

Motion Activity

Mosaic

Warping Parameters

Moving Region

Trajectory

Parametric Motion

39

MPEG-7 Terminology: MPEG-7 Terminology: Descriptor ValueDescriptor Value

• InstantiationInstantiation of a of a DescriptorDescriptor for a given data for a given data

set, or subset of that data setset, or subset of that data set

• Descriptor ValuesDescriptor Values are combined using a are combined using a

Description SchemeDescription Scheme to form a to form a DescriptionDescription

40

Motion ActivityMotion Activity

• Need to capture “pace” or Intensity of activityNeed to capture “pace” or Intensity of activity– ““High Action” chase scenes segmentsHigh Action” chase scenes segments

– ““Low Action” talking heads segmentsLow Action” talking heads segments

• Use Gross Motion Characteristics Use Gross Motion Characteristics – avoiding object segmentation, tracking etc.avoiding object segmentation, tracking etc.

41

INTENSITYINTENSITY

• Expresses “pace” or Intensity of ActionExpresses “pace” or Intensity of Action

• Uses scale of very low - low - medium - high - Uses scale of very low - low - medium - high -

very highvery high

• Extracted by suitably quantizing variance of Extracted by suitably quantizing variance of

motion vector magnitudemotion vector magnitude

42

SPATIAL DISTRIBUTIONSPATIAL DISTRIBUTION

• Captures the size and number of moving Captures the size and number of moving

regions in the shot on a frame by frame basisregions in the shot on a frame by frame basis

• Enables distinction between shots with Enables distinction between shots with one one

large regionlarge region in the middle ( e.g.,talking heads) in the middle ( e.g.,talking heads)

and shots with and shots with multiple small moving regionsmultiple small moving regions

(e.g.,aerial soccer shots)(e.g.,aerial soccer shots)

43

TEMPORAL DISTRIBUTIONTEMPORAL DISTRIBUTION

• Expresses fraction of the duration of each Expresses fraction of the duration of each level of activity in the total duration of the shotlevel of activity in the total duration of the shot

• Straightforward extension of the intensity of Straightforward extension of the intensity of motion activity to the temporal dimensionmotion activity to the temporal dimension

• A talking head, typically exclusively low A talking head, typically exclusively low activity, would have zero entries for all levels activity, would have zero entries for all levels except oneexcept one

44

DIRECTIONDIRECTION

• Expresses dominant direction if definable as Expresses dominant direction if definable as one of a set of eight equally spaced directionsone of a set of eight equally spaced directions

• Extracted by using averages of angle Extracted by using averages of angle (direction) of each motion vector(direction) of each motion vector

• Useful where there is strong directional motionUseful where there is strong directional motion

45

MPEG-7 Terminology: MPEG-7 Terminology: Description SchemeDescription Scheme

• Specifies Specifies structure and semanticsstructure and semantics of of

relationships between its componentsrelationships between its components• Components may be both Components may be both DescriptorsDescriptors and and

Description SchemesDescription Schemes– A A DescriptorDescriptor contains only basic data types, provided contains only basic data types, provided

by the by the Description Definition LanguageDescription Definition Language– A A DescriptorDescriptor does not refer to another does not refer to another DescriptorDescriptor

46

Description Scheme ExampleDescription Scheme Example

• Movie, temporally structured as scenes and Movie, temporally structured as scenes and shotsshots– Including textual Including textual descriptorsdescriptors at the scene at the scene

levellevel– Including color, motion and audio Including color, motion and audio descriptorsdescriptors

at the shot levelat the shot level

47

Description Schemes in MPEG-7Description Schemes in MPEG-7

• Creation and ProductionCreation and Production– Title, creator, classification, purpose of creationTitle, creator, classification, purpose of creation

• UsageUsage– Rights holders, access rights, publication, financial infoRights holders, access rights, publication, financial info

• MediaMedia– Storage format, AV content encoding, media identificationStorage format, AV content encoding, media identification

• Structural AspectsStructural Aspects– Color, texture, shape, motion, audio Color, texture, shape, motion, audio

• Conceptual AspectsConceptual Aspects– AV conceptual notionsAV conceptual notions

• Basic ElementsBasic Elements– Data types, math structures, schema toolsData types, math structures, schema tools

48

MPEG-7 Terminology: MPEG-7 Terminology: DescriptionDescription

• Consists of a Consists of a Description SchemeDescription Scheme and the set and the set

of of Descriptor ValuesDescriptor Values (instantiations) that (instantiations) that

describe the Datadescribe the Data• The The Description SchemeDescription Scheme may not be fully may not be fully

instantiated, depending upon completeness of instantiated, depending upon completeness of

the the Descriptor ValuesDescriptor Values set set

49

MPEG-7 Terminology: MPEG-7 Terminology: Description Definition Description Definition Language (DDL)Language (DDL)

• Language that enables creation of new Language that enables creation of new

Description SchemesDescription Schemes and and DescriptorsDescriptors

• Enables extension and modification of existing Enables extension and modification of existing

Description SchemesDescription Schemes

• Expresses relations, object orientation, Expresses relations, object orientation,

composition, partial instantiationcomposition, partial instantiation

50

DDL Logical ComponentsDDL Logical Components

• XML Schema structural language componentsXML Schema structural language components• XML Schema structural datatype componentsXML Schema structural datatype components• MPEG-7 specific extensionsMPEG-7 specific extensions

– Datatypes for matrices and arraysDatatypes for matrices and arrays– Datatypes for time point and durationDatatypes for time point and duration– Data value propagation (HeaderType)Data value propagation (HeaderType)

51

MPEG-7 SystemsMPEG-7 Systems

• Specifies Specifies functionalitiesfunctionalities such as preparation of such as preparation of MPEG-7 DescriptionsMPEG-7 Descriptions– Efficient transport/storageEfficient transport/storage– Synchronization of content and descriptionSynchronization of content and description– Development of conformant decodersDevelopment of conformant decoders

• Mechanism for providing multimedia content is Mechanism for providing multimedia content is considered part of a complete application and considered part of a complete application and lies outside the scope of the standardlies outside the scope of the standard

52

MPEG-7 TerminalMPEG-7 Terminal

• ObtainsObtains MPEG-7 data from transport MPEG-7 data from transport• ExtractsExtracts elementary streams from delivery layer elementary streams from delivery layer

– Undo transport/storage specific framing/multiplexingUndo transport/storage specific framing/multiplexing– Retain synchronization timingRetain synchronization timing

• ForwardsForwards elementary streams of individual access units elementary streams of individual access units to compression layerto compression layer

• DecodesDecodes– Schema streams describing data structureSchema streams describing data structure– Full or partial content description streamsFull or partial content description streams

• GeneratesGenerates user requested multimedia streams user requested multimedia streams• Feeds backFeeds back via delivery layer for transmission/storage via delivery layer for transmission/storage

53

MPEG-7 TerminalMPEG-7 Terminal

54

MPEG-7 DDLMPEG-7 DDL

With extensions, XML meets key requirementsWith extensions, XML meets key requirements• Datatype definitionDatatype definition• D and DS declarationD and DS declaration• Attribute declarationAttribute declaration• Typed referenceTyped reference• Content modelContent model• Inheritance/subclassing mechanismInheritance/subclassing mechanism• Abstract D and DSAbstract D and DS• DS inclusionDS inclusion

55

MPEG-7 VisualMPEG-7 Visual

• Specifies set of Specifies set of standardized Ds and DSsstandardized Ds and DSs• Mainly address Mainly address specificspecific features features

– Color, texture, motionColor, texture, motion• Often requires other low-level Ds or support Often requires other low-level Ds or support

elementselements– StructureStructure – grid layout, spatial coordinates – grid layout, spatial coordinates– ViewpointViewpoint – multiple view – multiple view– LocalizationLocalization – region locator – region locator– TemporalTemporal – time series, temporal interpolation – time series, temporal interpolation

56

MPEG-7 VisualMPEG-7 VisualStandardized DescriptorsStandardized Descriptors

• ColorColor– Color Space, Color Quantization, Dominant Color, Scalable Color Space, Color Quantization, Dominant Color, Scalable

Color Color Layout,Color Structure, Group of Picture ColorColor Color Layout,Color Structure, Group of Picture Color• TextureTexture

– Homogeneous Texture, Texture Browsing, Edge histogramHomogeneous Texture, Texture Browsing, Edge histogram• ShapeShape

– Region Shape, Contour Shape, Shape 3DRegion Shape, Contour Shape, Shape 3D• MotionMotion

– Camera Motion, Motion Trajectory, Parametric Motion, Camera Motion, Motion Trajectory, Parametric Motion, Motion ActivityMotion Activity

• Face Recognition, othersFace Recognition, others

57

MPEG-7 AudioMPEG-7 Audio

• Specifies set of Specifies set of standardized Ds and DSsstandardized Ds and DSs• Addresses four classes of audioAddresses four classes of audio

– Pure music, Pure speech, Pure sound effects, Arbitrary Pure music, Pure speech, Pure sound effects, Arbitrary soundtrackssoundtracks

• May address audio featuresMay address audio features– Silence, Spoken content, Timbre Sound effects, Melody, etcSilence, Spoken content, Timbre Sound effects, Melody, etc

• Often requires other low-level Descriptor Often requires other low-level Descriptor categoriescategories– Scalable Series – Scalable Series – ScalableSeries, SeriesofScalarType, etcScalableSeries, SeriesofScalarType, etc– Audio Description Framework – Audio Description Framework – AudioSampledType, AudioSampledType,

AudioWaveformEnvelopeTypeAudioWaveformEnvelopeType

58

MPEG-7 AudioMPEG-7 AudioStandardized DescriptorsStandardized Descriptors

• SilenceSilence– SilenceTypeSilenceType

• Spoken content Spoken content (from speech recognition)(from speech recognition)– SpokenContentSpeakerTypeSpokenContentSpeakerType

• Timbre Timbre (perceptual features of instrument sounds)(perceptual features of instrument sounds)– InstrumentTimbreType, HarmonicInstrumentTimbreType, InstrumentTimbreType, HarmonicInstrumentTimbreType,

PercussiveInstrumentTimbreTypePercussiveInstrumentTimbreType• Sound effectsSound effects

– AudioSpectrumBasisType, SoundEffectFeatureTypeAudioSpectrumBasisType, SoundEffectFeatureType• Melody ContourMelody Contour

– CountourType, MeterType, BeatTypeCountourType, MeterType, BeatType• Description Schemes utilizing these Descriptors are Description Schemes utilizing these Descriptors are

also definedalso defined

59

MPEG-7 Multimedia Description MPEG-7 Multimedia Description Schemes (MDS)Schemes (MDS)

• Specifies Specifies high-level frameworkhigh-level framework for generic for generic descriptions of all kinds of multimediadescriptions of all kinds of multimedia

• Contrasts with specific descriptions addressed Contrasts with specific descriptions addressed by Visual and Audio, (parts 3 and 4)by Visual and Audio, (parts 3 and 4)

• LevelsLevels– Basic elementsBasic elements– Content management & Content descriptionContent management & Content description

• Creation and production viewpointCreation and production viewpoint• MediaMedia• UsageUsage• Structural Aspects,Structural Aspects,• Conceptual AspectsConceptual Aspects

60

MDS Hierarchy: Levels and MDS Hierarchy: Levels and RelationshipsRelationships

61

MPEG-7 Reference SoftwareMPEG-7 Reference Software

• Reference implementationReference implementation of relevant MPEG-7 of relevant MPEG-7 StandardStandard– Experimentation software (XM) Experimentation software (XM)

• Creation of of D and DSs bitstreams with Creation of of D and DSs bitstreams with normative syntax rather than tool performancenormative syntax rather than tool performance

• Four categories of componentsFour categories of components– DDL parses and DDL validation parserDDL parses and DDL validation parser– Visual DescriptorsVisual Descriptors– Audio DescriptorsAudio Descriptors– Multimedia Description Schemes (MDS)Multimedia Description Schemes (MDS)

62

MPEG-7 ConformanceMPEG-7 Conformance

GuidelinesGuidelines and and proceduresprocedures for testing for testing implementations for conformanceimplementations for conformance

63

Possible MPEG-7 ApplicationsPossible MPEG-7 ApplicationsAbstract RepresentationAbstract Representation

DescriptionGeneration

MPEG7Description

MPEG7Coded

DescriptionEncoder Decoder

Search /QueryEngine

64

Standard EigenfacesStandard Eigenfaces

The eigenfaces for this database were approximated using a principal components analysis on a representative sample of 128 faces. Recognition and matching was subsequently performed using the first 20 eigenvectors. In addition, each image was then annotated (by hand) as to sex, race, approximate age, facial expression, and other salient features. Almost every person has at least two images in the database; several people have many images with varying expressions, headwear, facial hair, etc.

http://whitechapel.media.mit.edu/vismod/demos/facerec/basic.html

65

66

Face RecognitionFace Recognition

The system diagram above shows a fully automatic system for detection, recognition and model-based coding of faces for potential applications such

as video telephony, database image compression, and automatic face recognition. The system consists of a two-stage object detection and alignment

stage, a contrast normalization stage, and a Karhunen-Loeve (eigenspace) based feature extraction stage whose output is used for both recognition

and coding. This leads to a compact representation of the face that can be used for both recognition as well as image compression. Good-quality

facial images are automatically generated using approximately 100-bytes worth of encoded data. The system has been successfully tested on a

database of nearly 2000 facial photographs from the ARPA FERET database with a detection rate of 97%. Recognition rates as high as 99% have

been obtained on a subset of the FERET database consisting of 2 frontal views of 155 individuals.

http://whitechapel.media.mit.edu/vismod/demos/facerec/system.html

67

Photobook Photobook http://wasi.www.media.mit.edu/people/tpminka/photobook/http://wasi.www.media.mit.edu/people/tpminka/photobook/

• ToolTool for performing for performing queriesqueries on image on image databases based on image content. databases based on image content.

• Works by Works by comparing featurescomparing features associated with associated with images, images, not the imagesnot the images themselves. themselves.

• Features are parameter values of particular Features are parameter values of particular models fitted to each image. models fitted to each image.

68

http://whitechapel.media.mit.edu/people/tpminka/photobook/foureyes/seg.html

69

Texture ModelingTexture Modeling

70

This is an example of a Photobook search based on shape. The query image is in the upper left; the images in a small tools database are displayed in raster scan order of similarity

71

Content Retrieval using Image Content Retrieval using Image as the Queryas the Query

3. Spatial edge distribution

2. Spatial color distribution

4. Object shape

1. Color histogram

72

Movie ToolMovie Tool

Composea logical structurePreview

Detect

temporal / spatial keys

AnnotateMPEG-7

73

ReferencesReferences

• ICCE 2001 MPEG-7 Tutorial Session, 6/17/2001, Smith, ICCE 2001 MPEG-7 Tutorial Session, 6/17/2001, Smith, Manjunath, DayManjunath, Day

• MPEG 7 Main Page MPEG 7 Main Page http://www.darmstadt.gmd.de/mobile/MPEG7/http://www.darmstadt.gmd.de/mobile/MPEG7/

• IEEE Transactions on Circuit and Systems for Video IEEE Transactions on Circuit and Systems for Video Technology, Vol. 11, No. 6, Special Issue on MPEG-7Technology, Vol. 11, No. 6, Special Issue on MPEG-7

• Special Thanks to Dr. Manjunath of UCSB for providing Special Thanks to Dr. Manjunath of UCSB for providing a copy of his ICCE foils for use at our IEEE meetinga copy of his ICCE foils for use at our IEEE meeting

74