Semantic Web Application:

Music Retrieval

Ying Ding

SLIS, IU

Enabling Access to Sound Archives through Integration, Enrichment and Retrieval

http://www.elec.qmul.ac.uk/easaier/

The EASAIER Project

EASAIER - Enabling Access to Sound Archives through Integration, Enrichment and Retrieval

EU funded project, 30month duration (started May 2006)

Partners:

EASAIER - Goals Overcome problems for many digital sound archives

concerning online access sound materials and related media often separate searching audio content limited

EASAIER Framework Integration of Sound Archives Low level audio feature extraction (speech/music) Intelligent User Interface Enhanced Access Tools

looping, marking of audio sound source separation time and pitch scale modification

Semantic Search Evaluation

Semantics in EASAIER

Description of metadata using an ontology High-level metadata

e.g. title, author of an audio asset sources are databases, files in e.g. DC, MARC

Low-level metadata e.g. speech event occurs at timestamp xyz feature extractor tools

Semantic Search Search across variety of metadata Search across multiple archives Similarity Search Related content acquisition from the Web

The EASAIER System

Music Ontology Overview

Merging existing related ontologies

Developed by QMUL

Cover the major requirements

Widely-adopted

Four core MO components FRBR FOAF Event Timeline http://musicontology.com/

The Music Ontology: Timeline Ontology

Expressing temporal information, e.g. This performance happened the 9th of March, 1984 This beat is occurring around sample 32480 The second verse is just before the second chorus

The Music Ontology: Event Ontology

Event — An arbitrary classification of a space/time region This performance involved Glenn Gould playing the piano This signal was recorded using a XXX microphone located at that particular place This beat is occurring around sample 32480

The Music Ontology: FRBR & FOAF

FRBR – Functional Requirements for Bibliographic Records Work — e.g. Franz Schubert's Trout Quintet Manifestation — e.g. the "Nevermind" album Item — e.g. my "Nevermind" copy

FOAF – Friend of a Friend Person Group Organization

The Music Ontology – Music Production Concepts

On top of FRBR: MusicalWork, MusicalManifestation (Record, Track, Playlist, etc.), MusicalItem

(Stream, AudioFile, Vinyl, etc.)

On top of FOAF: MusicArtist, MusicGroup, Arranger, Engineer, Performer, Composer, etc. — all

these are defined classes: every person involved in a performance is a a performer...

On top of the Event Ontology: Composition, Arrangement, Performance, Recording

Others : Signal, Score, Genre, Instrument, ReleaseStatus, Lyrics, Libretto, etc.

The Music Ontology – Music Production Workflow

Low-level metadata is output in RDF using Music Ontology Audio Feature extractor Speech recognition service Emotion detection service

High-level metadata import DB Schema Mapping

e.g. D2R, Virtuoso RDF Views

Standardized Metadata import DC, MARC, METS, ...

Linked Data ? DBPedia, Geonames, ...

Metadata in RDF

Hotbed Database

Music Ontology

Querying

Publishing Extending

Instruments Taxonomy

Hotbed RDF

the Semantic Archivist

Query Interface

Sound AccesstoolsFeatures

Extraction,Visualization,...

Use Case: Archive Publication - HOTBED

1) editing the ontology

using WSMT editor to extend the ontologyMusic

Ontology

Music Ontology

Graphical Edit

Text Edit

2) performing tests on the new extension What are the instruments in my taxonomy ? Did i forget any kind of [pipe] ?

3)mapping Scottish Instruments to a general Instruments taxonomy

4) relating and publishing Hotbed Relate tables from hotbed to concepts from the MO Publish on the semantic web via the D2R tool

Mapping

The server offers a SPARQL end-point for external apps

RDF Publicationvia D2R tool

HotbedDatabase

Music Ontology

Mapping Metadata to the Music Ontologies

Title: File 2Author: Oliver Iredale SearlePerfomers: Katie PunterSource Type: AudioSource: File 2 Instrument: FluteInstrument occurrence timings: 0"-16"Time Signature: 4/4Beats per minute: 50Tonality: Bb major

Searle Testbed

:music a mo:Signal ;dc:title "File 2" ;dc:author "Oliver Iredale Searle" ;

:music-performance a mo:Performance ;

mo:recorded_as :music ;mo:composer :OliverIredaleSearle

;mo:instrument mo:flute ;mo:performer :KatiePunter ;mo:bpm 50 ;

mo:meter "4/4" ;mo:key #BFlatMajor.

:KatiePunter a foaf:Person .:ss1 a af:PersonPlaying;

af:person :KatiePunter;event:time [tl:onTimeLine :tl1234;

tl:beginsAt "PT0S"; tl:duration "PT16S"; ].

<xml version="1.0" encoding="UTF-8"><speech_retrieveResult> <speech_descriptor word="power" audio_material="c:/hotbed/performance/1004.wav" position_sec="10" duration_sec="5" confidence="89" /> </speech_retrieveResult></xml>

<http://www.myarchive.org/signal/1234/event/powerPT10S> a af:Text;

af:text "power";af:confidence "89";event:time [

a time:timeInterval;tl:onTimeline

<http://www.myarchive.org/signal-timeline/1234>;tl:beginsAtDuration "PT10S";tl:durationXSD "PT5S";

].

ALL web service output

Mapping Metadata to the Music Ontologies

<metadata type="audio"> <category name="vamp"> <feature name="beats" type="variablerate" description="Detected Beats" unit="N/A"> <data> <event idx="0" timestamp=" 0.0928" duration="0" label="224.69 bpm"/> </data> </feature> </category></metadata>

Vamp Output

event:time [a time:Instant ;tl:onTimeLine :tl898;

tl:at "PT0.0928S"; ];mo:bpm "224.69";

Mapping Metadata to the Music Ontologies

Built on top of OpenRDF Sesame 2.0

Query interfaces

Web Service (Servlet)

HTTP SPARQL Endpoint

Web Service provides predefined SPARQL query templates

Themes

Music, Speech, Timeline, Related media, Similarity

Dynamic FILTER constructs

Results in SPARQL Query Results XML Format

Interface for RDF metadata import using the Archiver application

RDF Storage and Retrieval Component

Enhanced Client

Web client

Related media

Double-click

Related media on the web (1)

Result search for author “Coltrane”

Track selection

Web related media search launched automatically according to the name of the author

Related media on the web (2)

Demo

http://www.elec.qmul.ac.uk/easaier/index-3.html

http://easaier.deri.at/demo/

Demo

Time and Pitch Scale Modification (demo) Sound source separation (demixing/remixing,

Noice reduction, etc.) (demo) Video time stretching (to slow down or speed

up images while retaining optimal sound) (demo)

Scenario 1 – Artist Search

Aggregation of music artist information from multiple web sources

Ontology based search: MusicBrainz data mapped to the MusicOntology MusicBrainz Web Service:

allows to retrieve artist URI by literal based search MusicBrainz RDF Dump:

retrieve RDF use SPARQL to perform queries (e.g. resolve relationships)

Web2.0 Mashups: Retrieve data (videos, images) from external sources

utilize RSS Feeds, APIs etc. from Youtube, LyricWiki, Google more accurate results using references from MusicBrainz RDF data

Scenario 1 – Artist Search

“Beatles”

<URI>

<URI>

WS

Interface

RDF Dumpprocess data...

Scenario 1 – Artist Search

Scenario 1 – Artist Search

Scenario 2 – Instrument Reasoning

Reasoning over HOTBED instrument scheme Ontologize data from HOTBED (Scottish Music Archive)

Usage of D2R to lift data from legacy DBs to RDF Ontologies:

MusicOntology Instrument Ontology (domain related taxonomy)

Subsumption reasoning: Retrieve instrument tree Search for persons that play an instrument

Subclass relations: resolve persons playing more specific instruments Example: Wind-Instrument < WoodWind < Flute

Scenario 2 – Instrument Reasoning

Example: Search for people playing instrument of type Woodwind

Demo 3 – Rules

Infer new knowledge with rules Domain Rule Sophisticated Query

Albums based on certain Band/Artist/Instrument

UseCase: The Velvet Underground discography Available information:

Membership durations Album release dates

„Founders“ of the band ? exist _artist, <_band, hasMember, _artist>, <_artist, onDuration, _duration> forall ?x, <_band, hasMember, ?x>, < ?x, onDuration, ?time> <?time, notBefore, _duration>

<_band, founder, _artist>

Albums & corresponding members

Demo 3 – RulesBasic

Information

Band Founder

Band Duration(Members & Albums)

Album Tracks