WP5.4 - Introduction Knowledge Extraction from Complementary Sources

This activity is concerned with augmenting the semantic multimedia metadata basis by analysis of complementary textual, speech and semi-structured data

Focus in first 12 months Joint work between DFKI, UEP and DCU on aligning event

extraction from textual football match reports with event recognition in video coverage of the same match

Focus in following 12 months Joint work between DFKI, UEP and DCU on the extension of

the event alignment work towards cross-media feature extraction (aligning low-level image/video features with events extracted in aligned textual and semi-structured data)

Joint work between DFKI, UEP, TUB and GET (cross-WP cooperation with WP3.3) on analyzing textual metadata in primary sources (OCR applied to text detected in images).

Text-Video Mapping in the Football Domain

Cooperation: DFKI, UEP, DCU Resources:

DFKI: SmartWeb Data Set (textual and tabular match reports) DFKI/UEP: Additional minute-by-minute textual match reports

(‚tickers‘) from other web resources DCU: Video Detectors (Crowd image detector, Speech-Band Audio

Activity, On-Screen Graphics Tracking, Motion activity measure, Field Line orientation, Close-up)

Textual and semi-structured data (tabular, XML files) are exploited as background knowledge in filtering the video analysis results and will possibly help in further improving the corresponding video analysis algorithms

Alignment of extracted events from unstructured textual data and from events that are provided by the semi-structured tabular data in the SmartWeb corpus (DFKI) with events that were detected by the video analysis results (DCU).

The SmartWeb Data Set as provided by DFKI is an experimental data set for ontology-based information extraction and ontology learning from text that has been compiled for the SmartWeb project.

The data set consists of: An ontology on football (soccer) that is integrated with foundational

(DOLCE), general (SUMO) and task-specific (discourse, navigation) ontologies.

A corpus of semi-structured and textual match reports (German and English documents) that are derived from freely available web sources. The bilingual documents are not translations, but are aligned on the level of a particular match (i.e. they are about the same match).

A knowledge base of events and entities in the world cup domain that have been automatically extracted from the German documents.

For the purposes of the experiment described here we were mostly interested in the events that are described by the semi-structured data.

Resources

SmartWeb Data Example

Framework for event detection in broadcast video of multiple different field sports as provided by DCU

Video detectors used by DCU Crowd image detector Speech-Band Audio Activity On-Screen Graphics Tracking Motion activity measure Field Line orientation Close-up

DCU: Video Analysis Data

0

100

200

300

400

500

600

700

<0.1-0.2> <0.2-0.3> <0.3-0.4> <0.4-0.5> <0.5-0.6> <0.6-0.7> <0.7-0.8> <0.8-0.9> <0.9-1>

01000200030004000500060007000

<0-0

.1)

<0.1

-0.2

)

<0.2

-0.3

)

<0.3

-0.4

)

<0.4

-0.5

)

<0.5

-0.6

)

<0.6

-0.7

)

<0.7

-0.8

)

<0.8

-0.9

)

<0.9

-1)

<1>

Řada1

0200400600800

10001200

<0-0

.1)

<0.1

-0.2

)

<0.2

-0.3

)

<0.3

-0.4

)

<0.4

-0.5

)

<0.5

-0.6

)

<0.6

-0.7

)

<0.7

-0.8

)

<0.8

-0.9

)

<0.9

-1)

<1>

Řada1

crowd

audio

confidence

0200400600800

10001200140016001800

<0-0

.1)

<0.1

-0.2

)

<0.2

-0.3

)

<0.3

-0.4

)

<0.4

-0.5

)

<0.5

-0.6

)

<0.6

-0.7

)

<0.7

-0.8

)

<0.8

-0.9

)

<0.9

-1)

<1>

Řada1

Visual_motion

DFKI/UEP: Extraction of Tickers

Ard.deLigalive.de bild.de

•

Minute-by-minute reports from different Web resources

Shallow Processing with Unification and Typed Feature Structures (SProUT) tool for multilingual shallow text processing and information extraction

SProUT java web service that takes the minute-by-minute reports as an input, parses them and extracts a new XML file for each minute of a particular match

Information Extraction from Text

Information Extraction with DFKI Tool „SProUT“

Information Extraction Results (SProUT)

Aligning and Aggregation of Textual Events

Events alignment from various tickers

alignment

Example: minute 40

Tabular Reports

+ video event detection data (features) from DCU

Min

ute

-by-

min

ute

rep

ort

s

VIDEO – TEXTUAL DATA TIME ALIGNMENT

CROSS-MEDIA FEATURE EXTRACTION

Data aggregation for later use

Match vs Video Time

Freekick evaluation

Time differences tracking

Possible OCR on video

Purpose: Cross-Media features describe information that occurs in textual/semi-structured data as well as in video data and can therefore be used as additional support in video analysis.

Goal: Use video detectors aligned with events extracted from text/semi-structured data as cross—media features

Example:

Cross-media Features

Summary Extracted: 1200 events, 45 event-types After alignment: 850 events describing

five matches from World Cup 2006 Final 170 events per game on average Cross-media descriptors for every

event-type

Future plans In WP5.4.1 continue work on mapping between

results of video analysis and complementary resource analysis in the following way: Use extracted image descriptors from training data (video

+ aligned text extraction) for the classification of fine-grained events in test data (i.e. other videos) -- all based on minute-by-minute alignment

Cooperate with TUB in Video OCR to help time video-text alignment

WP5.4.2 Images and text as mutually complementary resources

WP5.4.3: Image retrieval based on enhanced query processing and complementary resource analysis

Apart from identifying individual events, it might be useful to find out about general statistical dependencies (associations) among types of events

Initial experiments carried out on a single type of resource – structured data In the future, events extracted from text and video could be considered as

well Use of LISp-Miner tool (UEP)

Data mining procedure 4ft-Miner mines for various types of association rules and conditional association rules

Potential application: Discovering new relationships to be inserted into the domain ontology or knowledge base,

Mining over Football Match Data: Seeking Associations among Explicit and Implicit

Events

Joint Work Example