Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets:...

13
1 Introduction (Motivation) In 2015, a total of 728 millions of public pictures were uploaded to Flickr Such large amount of user-generated data makes multimedia indexing and retrieval a more challenging task However, it also opens new opportunities for development of novel and more efficient tools

Transcript of Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets:...

Page 1: Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

1

Introduction(Motivation)

In2015,atotalof728millionsofpublicpictureswereuploadedtoFlickr

Suchlargeamountof user-generateddatamakesmultimediaindexingandretrievalamorechallengingtask

However,italsoopensnewopportunitiesfordevelopmentofnovelandmoreefficienttools

Page 2: Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

2

Introduction(Motivation)User-generated multimedia contents depictindividual experiences or collective activities

WhatisanEvent?

Arealworldhappening toWho?,What?,When?andWhere?

Aneventisplannedbypeopleattendedbypeopleandrelatedmediaarealsocapturedbypeople

Personalexperiences

Collectiveactivities

Page 3: Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

3

EventDetectioninImages:State-of-the-art

VisualInformation

Metadata(tags,GPSinformation

etc.)

Visual+Metadata

Page 4: Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

4

BenchmarkDatasets:State-of-the-art

Currentdatasetsfor

eventdetectioninimages

lownumberofimages(e.g.,EIMM[1],Cultural

eventrecognitiondatabase[3])

limitedvarietyofevents/eventclasses(e.g.,EiMM [2]andSED2013

database[2])

Unbalancedeventclasses(e.g., EiMM [1]andSED2013[2])

1. R.Mattivi etal..Exploitationoftimeconstraintsfor(sub-)eventrecognition.InProceedingsofthe2011jointACMworkshoponModelingandrepresentingevents,pages7(12).ACM,2011..

2. T.Reuteretal..Socialeventdetectionatmediaeval2013:Challenges,datasets,andevaluation.InMediaEval Workshop,2013..3. S.Escalera etal..ChaLearn LookingatPeople2015:ApparentAgeandCulturalEventRecognitionDatasetsandResults,ICCV2015

Page 5: Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

5

USED:AlargeScaleSocialEventDetectionDatasetAlargecollectionofimages

Covers14differenteventsclasses

AbalanceddatasetEqualnumberofimagesineachclass(35,000)

Event-classesinUSEDDataset

Page 6: Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

6

USED:AlargeScaleSocialEventDetectionDataset

DiversityincontentsIndoorVs.outdoorGrouppicturesVs.SingleportraitImagesofkey-momentsinaneventMulti-culturalOutliersandborderlinecasesaremanuallyremoved

Somesampleimagesfromweddingclass

Page 7: Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

7

USED:AlargeScaleSocialEventDetectionDataset

USED490,000 Eventrelated

imagesdepictinga widevarietyof

events

Page 8: Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

8

Comparisonswithstate-of-the-artdatasets

ExistingdatasetsforEventDetectionCulturalEventDetectionDatasetEiMMSED

DatasetName #Event-classes Total Images Minimagesinaclass

Max.images inaclass

EiMM 8 (socialevents) 13219 795 2253

SED 7 82213 342 71556

CulturalEvents 50 11776 180-200(Avg.) 180-200(Avg.)

USED 14 490000 35000 35000

Comparisons ofUSEDwithotherDatasets

Page 9: Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

9

ExperimentalValidationofUSED

DISCOVERINGEVENTSFROMSINGLEPICTURESUSINGACONVOLUTIONALNEURALNETWORK

Page 10: Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

10

Validation/ExperimentalSetup

Fine-tuningCNN

Classification

Pre-training

ParametersofaCNN(Alexnet)pre-trainedonImageNet dataset

[NIPS2012]

Fine-tunedonnewlycollecteddatasets

Reduced overalllearningrateIncreasedlearningrateof

newlayerMomentum=.9

WeightDecay=.0005

Page 11: Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

11

PreliminaryResultsDataset

USED

Event Type Accuracy EventType Accuracy

Concert 74.20% Conference 75.70%

Graduation 66.43% Exhibition 58.54%

Meeting 78.70% Fashion 65.43%

MountainTrip 67.00% Protest 74.58%

Picnic 54.42% Sports 72.24%

Sea-holiday 74.24% Theater 51.90%

Ski-holiday 48.00%

Wedding 51.00%

ResultsonUSEDdataset

DataAssemblageTrainingset=20,000imagesperclassValidationset=7000perclassTestset=7000imagesperclass

Page 12: Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

12

ComparisonsofaCNNtrainedonUSEDwithBaselineApproaches

ComparisonwithRosani etal.,[IEEETMM2015]

EiMMDataset SEDDatasetOurApproach 71.54 59.42BaselineApproach 38.8 31.15

0

10

20

30

40

50

60

70

80Ac

curacy(%

)

A.Rosani,G.Baoto,F.G.B.DeNatale,“EventMask:agame-basedframeworkforEvent-saliencyidentificationinImages”,IEEETransactionsonMultimedia2015

Page 13: Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

13

USED:ALarge-scaleSocialEventDetectionDataset

490,000 Event-relatedimages, 14differentevent-classes,35,000imagesper

class

ENJOY USED!


The Materials Data Facility - Northwestern University · 5 Publication APIs • Identify datasets with persistent identifiers (e.g. DOI) • Describe datasets with appropriate metadata,

The Materials Data Facility - Northwestern University · 5 Publication APIs • Identify datasets with persistent identifiers (e.g. DOI) • Describe datasets with appropriate metadata,

Metadata Considerations Implementing Administrative and Descriptive Metadata for your digital images 1.

Metadata Considerations Implementing Administrative and Descriptive Metadata for your digital images 1.

BLT: Balancing Long-Tailed Datasets with Adversarially-Perturbed … · 2020. 11. 8. · BLT: Balancing Long-Tailed Datasets with Adversarially-Perturbed Images Jedrzej Kozerawski1,

BLT: Balancing Long-Tailed Datasets with Adversarially-Perturbed … · 2020. 11. 8. · BLT: Balancing Long-Tailed Datasets with Adversarially-Perturbed Images Jedrzej Kozerawski1,

Metadata Unmanaged API - blog.monstuff.comblog.monstuff.com › archives › images › Metadata_Umanaged_API.pdf · Metadata API Page 11 1 Overview of the Metadata API This document

Metadata Unmanaged API - blog.monstuff.comblog.monstuff.com › archives › images › Metadata_Umanaged_API.pdf · Metadata API Page 11 1 Overview of the Metadata API This document

Work Session on Statistical Metadata (METIS) Terri Mitton, 10 March 2010 Publishing Standards for Datasets and Data Tables.

Work Session on Statistical Metadata (METIS) Terri Mitton, 10 March 2010 Publishing Standards for Datasets and Data Tables.

VoID: Metadata for RDF Datasets

VoID: Metadata for RDF Datasets

Managing Imagery using Mosaic Datasets - Esri...•The Image Service is the primary information model for imagery on the web.-Single Images-Large collections using mosaic datasets•Puts

Managing Imagery using Mosaic Datasets - Esri...•The Image Service is the primary information model for imagery on the web.-Single Images-Large collections using mosaic datasets•Puts

1. Data providers deliver metadata records that describe their datasets through OGC catalogue services for the web. Each metadata record uses keyword concepts.

1. Data providers deliver metadata records that describe their datasets through OGC catalogue services for the web. Each metadata record uses keyword concepts.

Semantic Segmentation Datasets for Resource Constrained ...cvit.iiit.ac.in/images/ConferencePapers/2019/Semantic...Semantic Segmentation Datasets for Resource Constrained Training

Semantic Segmentation Datasets for Resource Constrained ...cvit.iiit.ac.in/images/ConferencePapers/2019/Semantic...Semantic Segmentation Datasets for Resource Constrained Training

The Vogue Archive - Leveraging Images and Metadata for Fashion and Cultural Research

The Vogue Archive - Leveraging Images and Metadata for Fashion and Cultural Research

Technologies and Tools to Search Images with Images · – Text-based – Image data is a passive component of the query • Model 2: Metadata-tagged Images – Image Metadata associated

Technologies and Tools to Search Images with Images · – Text-based – Image data is a passive component of the query • Model 2: Metadata-tagged Images – Image Metadata associated

Using HTML Metadata to Retrieve Relevant Images from the World Wide Web

Using HTML Metadata to Retrieve Relevant Images from the World Wide Web

Agenda · 2020. 10. 1. · • Metadata about the overall exchange catalogue • Metadata about the individual datasets • Metadata about the support files that make up the package

Agenda · 2020. 10. 1. · • Metadata about the overall exchange catalogue • Metadata about the individual datasets • Metadata about the support files that make up the package

CASE STUDY: MEDIA & METADATA - Clipsource€¦ · qualified metadata and richer content, such as trailers, clips and images. Internet users simply expect deeper information, and this

CASE STUDY: MEDIA & METADATA - Clipsource€¦ · qualified metadata and richer content, such as trailers, clips and images. Internet users simply expect deeper information, and this

Geospatial One Stop Modules Two and Three. Module 2 Inventory/Document existing Federal agency framework datasets and publish metadata to clearinghouse.

Geospatial One Stop Modules Two and Three. Module 2 Inventory/Document existing Federal agency framework datasets and publish metadata to clearinghouse.

Faceted Metadata in Image Search & Browsing Using Words to Browse a Thousand Images

Faceted Metadata in Image Search & Browsing Using Words to Browse a Thousand Images

D1.4.2 Metadata Management and Reasoning with Networked Ontologies …neon-project.org/web-content/images/Publications/neon... · 2010. 8. 11. · D1.4.2 Metadata Management and Reasoning

D1.4.2 Metadata Management and Reasoning with Networked Ontologies …neon-project.org/web-content/images/Publications/neon... · 2010. 8. 11. · D1.4.2 Metadata Management and Reasoning

A journey through time: Digitizing, processing and ...around 400’000 aerial images (starting 1926) in the archives 329’885 images digitized & metadata collected 45’689 images

A journey through time: Digitizing, processing and ...around 400’000 aerial images (starting 1926) in the archives 329’885 images digitized & metadata collected 45’689 images

From Several Datasets to One Graph - Society of American ... · From Several Datasets to One Graph designing a prototype for a graph visualization of archival metadata from different

From Several Datasets to One Graph - Society of American ... · From Several Datasets to One Graph designing a prototype for a graph visualization of archival metadata from different

Generation of real-time SDTM datasets and metadata through a generic SDTM converter mechanism CDISC (CDASH/SDTM) integration into OC/RDC Peter Van Reusel.

Generation of real-time SDTM datasets and metadata through a generic SDTM converter mechanism CDISC (CDASH/SDTM) integration into OC/RDC Peter Van Reusel.