Final Report on Integration - Tosca-mp€¦ · Final Report on Integration Deliverable D6.6...

© TOSCA-MP consortium: all rights reserved page i

Final Report on

Integration

Deliverable D6.6

TOSCA-MP identifier: TOSCAMP-D6.6-IRT_FinalReportOnIntegration_v10_Public.docx

Deliverable number: D6.6

Author(s) and company: Ronald Mies, Matthias Elser, Gordana Polanec-Kutija, Peter Altendorf (IRT)

Carlos Ruiz, Víctor Méndez, Jesús Contreras (PLY)

Albert Hofmann, Rene Schuster, Werner Bailer (JRS)

Patrick Ndjiki-Nya, Sebastian Gerke (HHI)

Frank Gläser, Michael Weber (DTO)

Mike Matton (VRT)

Internal reviewers: Albert Hofmann (JRS)

Work package / task: WP6

Document status: FINAL

Confidentiality: Public

Version Date Reason of change

0.1 2014-02-14 document created, structure proposed

0.2 2014-03-07 first inputs to document

0.3 2014-03-20 first consolidated version

0.4 2014-03-28 consolidated version for internal review

1.0 2014-04-03 final version

Version of 2014-04-03 D6.6. Final Report on Integration

© TOSCA-MP consortium: all rights reserved page ii

Acknowledgement: The research leading to these results has received funding from the European Union's Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 287532.

Disclaimer: This document does not represent the opinion of the European Community, and the European Community is not responsible for any use that might be made of its content.

This document contains material, which is the copyright of certain TOSCA-MP consortium parties, and may not be reproduced or copied without permission. All TOSCA-MP consortium parties have agreed to full publication of this document. The commercial use of any information contained in this document may require a license from the proprietor of that information.

Neither the TOSCA-MP consortium as a whole, nor a certain party of the TOSCA-MP consortium warrant that the information contained in this document is capable of use, nor that use of the information is free from risk, and does not accept any liability for loss or damage suffered by any person using this information.


© TOSCA-MP consortium: all rights reserved page iii

Table of Contents

Table of Contents ................................................................................................................................. iii

List of Figures ........................................................................................................................................ v

List of Tables ........................................................................................................................................ vi

1 Executive Summary ......................................................................................................................... 7

2 Introduction ....................................................................................................................................... 8

2.1 Purpose of this Document .......................................................................................................... 8

2.2 Scope of this Document.............................................................................................................. 8

2.3 Status of this Document.............................................................................................................. 8

2.4 Related Documents .................................................................................................................... 8

3 TOSCA-MP system design .............................................................................................................. 9

3.1 The Logical System Design – a recap ........................................................................................ 9

3.2 The Technical System Design – a recap .................................................................................. 10

3.3 Selection of the implementation platform ................................................................................. 11

4 The integration process ................................................................................................................. 14

4.1 Realization of an executable Business Process Model ............................................................ 14

4.2 Technical integration of services in the PoC ............................................................................ 15

4.3 Additional TOSCA-MP-specific developments, supporting the system integration .................. 17 4.3.1 Handling of essence and metadata in a distributed environment ........................................................................ 17 4.3.2 Adding Process Logic to the BPM ....................................................................................................................... 18

5 Integrated Proof of Concept realisation ....................................................................................... 19

5.1 DRF ........................................................................................................................................... 19

5.2 Content Analysis services......................................................................................................... 19 5.2.1 Visual analysis .................................................................................................................................................... 19 5.2.2 Audio/text analysis .............................................................................................................................................. 20 5.2.3 Multimodal analysis ............................................................................................................................................. 20 5.2.4 Low-level feature extraction and utilities .............................................................................................................. 20

5.3 Additional services .................................................................................................................... 20 5.3.1 Index Service / Search service ............................................................................................................................ 20 5.3.2 Benchmarking service ......................................................................................................................................... 21 5.3.3 Export & import Service to / from Metadata Schema Dependent Repository Extension ....................................... 22 5.3.4 Verify Quality Analysis service ............................................................................................................................. 22 5.3.5 Audio Extraction service ...................................................................................................................................... 23 5.3.6 Basket Sharing service ........................................................................................................................................ 24

5.4 MPMF ....................................................................................................................................... 24 5.4.1 Flexible workflow configuration ............................................................................................................................ 25 5.4.2 BPMN implementation of the CATM .................................................................................................................... 26 5.4.3 Generic service process ...................................................................................................................................... 26 5.4.4 Generic service and DRF handler ........................................................................................................................ 30

6 Graphical User Interfaces and applications ................................................................................ 31

6.1 Control & Configuration ............................................................................................................ 31 6.1.1 Functionality ........................................................................................................................................................ 31 6.1.2 Related services .................................................................................................................................................. 35


© TOSCA-MP consortium: all rights reserved page iv

6.1.3 Role in the overall workflow ................................................................................................................................. 35

6.2 AVDP Viewer ............................................................................................................................. 36 6.2.1 Functionality ........................................................................................................................................................ 36 6.2.2 Related services .................................................................................................................................................. 37 6.2.3 Role in the overall workflow ................................................................................................................................. 37

6.3 Interactive manual annotation ................................................................................................... 38 6.3.1 Functionality ........................................................................................................................................................ 38 6.3.2 Related services .................................................................................................................................................. 39 6.3.3 Role in the overall workflow ................................................................................................................................. 39

6.4 Search ....................................................................................................................................... 39 6.4.1 Functionality ........................................................................................................................................................ 39 6.4.2 Related services .................................................................................................................................................. 41 6.4.3 Role in the overall workflow ................................................................................................................................. 41

6.5 Football match summary ........................................................................................................... 41 6.5.1 Functionality ........................................................................................................................................................ 41 6.5.2 Related services .................................................................................................................................................. 42 6.5.3 Role in the overall workflow ................................................................................................................................. 42

7 Conclusions..................................................................................................................................... 43

8 Glossary ........................................................................................................................................... 44

9 Annexes ........................................................................................................................................... 45

9.1 Logical System Design – comprehensive version .................................................................... 46

9.2 Configurable Analysis Task Model – comprehensive version .................................................. 47

9.3 Overview of the services within the TOSCA-MP PoC ............................................................... 48

9.4 Sample XML view of the MPMF log file .................................................................................... 54


© TOSCA-MP consortium: all rights reserved page v

List of Figures

Figure 1: Simplified version of the Logical System Design ........................................................................ 9

Figure 2: Technical System Design .......................................................................................................... 10

Figure 3: Test project implemented in selected SOA environments to gain hands-on experience ......... 12

Figure 4: Typical workflow as realized for test project on selected SOA environments........................... 12

Figure 5: Iteration steps towards the TOSCA-MP Proof of Concept ........................................................ 14

Figure 6: Simplified representation of the Configurable Analysis Task Model ......................................... 15

Figure 7: SoapUI interface, showing Test Suite, Test Steps and results for the Concept Detection service ...................................................................................................................................................... 16

Figure 8: Handling of essence and metadata in the distributed PoC ....................................................... 17

Figure 9: Implementation of the benchmarking service ........................................................................... 22

Figure 10: Job queue of the verification service ....................................................................................... 22

Figure 11: Quality Summary user interface used for verifying quality annotations .................................. 23

Figure 12: Shared content basket service data model ............................................................................. 24

Figure 13: MPMF detailed architecture within the TOSCA-MP PoC ........................................................ 25

Figure 14: Service object XML structure .................................................................................................. 26

Figure 15: Generic service process within the MPMF .............................................................................. 27

Figure 16: Generic service process including error handling ................................................................... 28

Figure 17: Log file example of the JBoss LogEvent ................................................................................. 28

Figure 18: Log file object XML schema (see annex 9.4 for a sample log file) ......................................... 29

Figure 19: The DRF Tab ........................................................................................................................... 31

Figure 20: Tab „Process Start Items“ – Select process, configuration and start an analysis process for each selected content item ....................................................................................................................... 32

Figure 21: Tab „Process Start Content baskets“ – Select process, configuration and start an analysis process for each item in each selected content basket ........................................................................... 33

Figure 22: Tab „Benchmarking“: Allows to set up and start analysis processes for selected content baskets and the execution of a benchmarking process ........................................................................... 34

Figure 23: Benchmarking workflow .......................................................................................................... 34

Figure 24: Tab „Process Monitoring“: Shows the status of all running processes ................................... 35

Figure 25: User interface of the AVDP viewer.......................................................................................... 36

Figure 26: AVDP viewer, with some views zoomed in ............................................................................. 37

Figure 27: Navigation using the events in the table ................................................................................. 37

Figure 28: User Interface 1 of manual annotation tool ............................................................................. 38

Figure 29: User Interface 2 of manual annotation tool ............................................................................ 39

Figure 30: Semantic Multimedia Annotation and Search interface .......................................................... 40

Figure 31: Visual clustering of the search results .................................................................................... 41

Figure 32: User interface of sports events visualization ........................................................................... 42


© TOSCA-MP consortium: all rights reserved page vi

List of Tables

Table 1: Layer in the Technical System Design ...................................................................................... 10

Table 2: SOA platforms that were evaluated for use in TOSCA-MP ....................................................... 11


© TOSCA-MP consortium: all rights reserved page 7

1 Executive Summary

This document describes the integrated Proof of Concept (PoC) as it was realised by TOSCA-MP based on the system architecture (as described in D6.3) following the principles of Service-oriented Architecture (SOA).

The integration process focussed on the realisation of the so-called Configurable Analysis Task Model as a means of combining the individual formal task models of several usage scenarios in a single analysis workflow, to avoid unnecessary duplication at runtime.

The technical integration of the components developed in WP2, WP3, WP5 and WP6 was an iterative process which used formal descriptions of all components and their interfaces to facilitate extensive testing before the actual integration of the web service-based components into the PoC. The technical components of which the PoC is composed are described as well; for components developed in WP2, WP3 and WP5, an overview is given, focussing on the relevant aspects with respect to integration in the PoC; for details the reader is kindly referred to deliverables of the respective WPs. Specifically, the web interfaces for these components are addressed here, as these form the technical basis for their integration in the PoC.

The Metadata Production Management Framework (MPMF) as the core component for the integration of components as services, and for orchestration and execution of media analysis processes, is described in detail. It has been realised based on the Red Hat JBoss® Enterprise Application Platform, and contains a multitude of TOSCA-MP specific solutions for a flexible workflow configuration, pre-processing steps (before the actual invocation of analysis services), process logic and an extensive error handling and logging. For the handling of essence and metadata in the distributed PoC, a solution based on URL-exchange between MPMF and analysis services was implemented.

The Graphical User Interfaces (GUI) that provide the means for various users to interact with the system, either to control or configure it (administration) or to actually use it (search, annotate etc.) are presented. Each GUI makes use of suitable services to provide its functionality and by doing so serves the realisation of the project’s business goals. For each GUI it is described, how it fits in the respective workflow.

The integrated PoC is a fully functional prototype which has already been used on multiple occasions for demonstration and dissemination purposes (e.g. during the project’s field trials). The project partners aim at having it available for a year after the project’s finalisation, to facilitate further dissemination of the results.



2 Introduction

2.1 Purpose of this Document

The purpose of this document is to describe the final results of the integration process leading to the TOSCA-MP Proof of Concept.

2.2 Scope of this Document

This deliverable describes the TOSCA-MP integrated Proof of Concept, as well as the integration process itself and specifically gained knowledge during the process. Starting point for the descriptions in this deliverable is the System Design as described in D6.3, which is briefly recapitulated.

The main process of integration is described, as well as the technical integration of the components developed in the project and the way that the system’s functionality is being made available to end users by means of the Graphical User Interfaces.

This document does not give a detailed overview of the components that were developed in WP2, WP3 and WP5; the reader is kindly referred to the deliverables from the respective work packages (see section 2.4).

2.3 Status of this Document

This document is the final public version of the TOSCA-MP integrated Proof of Concept and concludes the work in WP6 of the project.

2.4 Related Documents

This document relates to the following existing TOSCA-MP deliverables:

D2.3 “Speech/visual metadata extraction, semantic enrichment and linking and genre adaptive processing v3”

D3.3 “Networked Media Search Engine, Visualization, and User Feedback v2”

D3.4 “Networked Media Search Engine, Visualization, and User Feedback v3”

D4.1 “Relevant tasks in the a/v media production workflow”

D5.1 “Overall architecture, interfaces and protocols of the Distributed Repository Framework”, version 08

D6.1 “Usage Scenarios”

D6.2 “Requirements”

D6.3 “System Design”

D6.5 “Proof-of-concept Integration”

Corresponding cross references are given in the respective sections.



3 TOSCA-MP system design

For the realization of the TOSCA-MP Proof of Concept (PoC) a system architecture had been commonly developed by all partners, in which all components were combined in a unified platform. This system design was documented in D6.3 and was based on the Usage Scenarios and the Requirements (described in D6.1 and D6.2 respectively). The system design describes connections and relationships between all identified components. Based on the system design, the PoC was realized by all partners cooperatively in WP6.

For ease of reference, the TOSCA-MP system design is briefly described in the following two sections; please refer to D6.3 for more details.

3.1 The Logical System Design – a recap

Figure 1: Simplified version of the Logical System Design

The Logical System Design (LSD) of TOSCA-MP provides a high-level definition of the platform: it shows an overview of the four main components in TOSCA-MP, how they are connected and which kind of data is exchanged between them. Figure 1 depicts a simplified version where only the four main system components and its connections are shown; a comprehensive version with more detailed information can be found in Annex 9.1.

The four main system components are:

Metadata Production Management Framework (MPMF) which orchestrates the functions and services

Distributed Repository Framework (DRF) which manages networked distributed local repositories and provides the possibility of integrating third party repositories

Services which provide functionalities (specifically annotations or feature extraction) to perform the tasks in TOSCA-MP

Graphical User Interfaces (GUI) to control / configure the system and review the results of user tasks



3.2 The Technical System Design – a recap

In addition to the Logical System Design a more technical view on the TOSCA-MP System is given here. Based on the principles of a Service Oriented Architecture (SOA) the Technical System Design (TSD) was developed. The TSD, as shown in Figure 2, specifically illustrates the foreseen technical integration based on Web Services.

Figure 2: Technical System Design

The general architecture can be seen as a multi-layer architecture. Having a top down view, the architecture can be described having layers as shown in Table 1.

Table 1: Layer in the Technical System Design

Layer Description

GUI / Application

Graphical User Interfaces (GUIs) form the top layer of the Technical System Design.

The GUIs provide access to the TOSCA-MP system. A GUI consists of one or multiple applications. The applications interact with the services inside the Orchestration Layer provided by the Metadata Production Management Framework (MPMF) and the Distributed Repository Framework (DRF).

Specifically, GUIs have been developed for visualization and annotation techniques, for controlling and configuring the system and for semantic search (see chapter 6).

Orchestration The Orchestration Layer is located below the Application Layer. It builds the wrapping layer of the

Service Layer and the

Metadata Production Management Framework (MPMF).

The core of the layer compound is built by the MPMF, responsible for the exchange of control data and metadata. The orchestration of services is defined inside this MPMF, as so-called processes. Processes can be used by the Application Layer to invoke the required business logic. A process engine inside the MPMF is responsible for the execution of the processes.

Processes provide their functionality to the Application Layer by Web service interfaces in the Service Layer. The Service Layer provides access to and control of the different components in the TOSCA-MP system by Web service interfaces. The realization of the MPMF is discussed in detail in section 5.4.

Component / The Component Layer, located below the Orchestration Layer, contains the actual components developed in TOSCA, like metadata generation and search



Persistence components, benchmarking components and the Distributed Repository Framework (DRF). Each of these components provides its functionality defined as a service to the Orchestration Layer by Web services interfaces in the Service Layer. The components in this layer are described in sections 5.1, 5.2 and 5.3.

Data The Data Layer forms the bottom layer of the Technical System Draft. It handles the content (essence & metadata) exchange between the DRF and the components or the GUIs.

3.3 Selection of the implementation platform

As a prerequisite the project from the start aimed at using SOA principles to realize the integrated PoC. For the realisation of the Orchestration Layer in the Technical System Design and the MPMF (see Figure 2) an appropriate SOA software environment had to be selected.

Table 2: SOA platforms that were evaluated for use in TOSCA-MP

Manufacturer Product Open source /

commercial

Software AG webMethods 8.2 commercial

Tibco Software ActiveMatrix Service Bus commercial

Oracle Sun GlassFish Enterprise Service Bus commercial

Progress Software Sonic ESB

Artix ESB

commercial

commercial

IBM WebSphere 7.0 commercial

Fuse Source Fuse ESB open source

WSO2 WSO2 Enterprise Service Bus open source

Red Hat JBoss ESB open source

MuleSoft Mule ESB open source

OW2 Consortium Petals ESB open source

Activiti org Activiti open source

Based on an Internet-search combined with study of market analysis documentation (e.g. from Forrester Research) an initial list of SOA platforms in the market was established, see Table 2. These platforms were evaluated in detail, based on the respective available documentation, with respect to:

License (commercial / open source)

Software - / hardware dependencies

Standards / protocols / interfaces

Adapter (Bindings)

System specifications (operating system, database, process engine)



Figure 3: Test project implemented in selected SOA environments to gain hands-on experience

Figure 4: Typical workflow as realized for test project on selected SOA environments

Based on this initial overview, a short list with 6 candidates was generated (marked blue in Table 2). These systems were analyzed in detail with respect to their ease of installation, features, performance, usability, extendibility and general potential for deployment in TOSCA-MP as well as checked against project specific requirements (e.g. support for SOAP & REST, BPMN2.0). For four of these environments (Activity org / Activity, OW2 Consortium / Petals ESB, RedHat / JBoss ESB and IBM / WebSphere ESB) also a small test project was realized for each of the software platforms (see Figure 3 and Figure 4) to inspect the respective process engine in practice and get hands-on experience in addition to the products’ documentation. For the remaining two SOA environments no (free) test software was available and a test project could not be implemented. Detailed evaluation results have been made available to the project partners.

Based on this evaluation, the Red Hat JBoss® Enterprise Application Platform1 (version 5.3) was

chosen as a basis for the realization of the MPMF within TOSCA-MP. JBoss2 provides a modular

architecture with much functionality rendering it very flexible. The platform is available as open source software, is Java based and comes with extensive documentation, allowing for efficient combination and

1 http://www.jboss.org/products/eap

2 The JBoss website: http://www.jboss.org/

http://www.jboss.org/products/eap

http://www.jboss.org/



extension with the specific software components which were developed to realize the MPMF. Red Hat offers added value on top of the open source platform e.g. in the form of regular bug fixes and support contracts for JBoss users, allowing for detailed and specific help with implementations.

The appropriate JBoss software modules were acquired, installed and configured by IRT to realize an orchestration system which was made available as an on-line platform. Section 5.4 describes how the MPMF has been realized based on the JBoss platform and which TOSCA-specific components allowing for handling of process logic, DRF communication and last but not least bilateral handling of the content analysis services, were developed and integrated in the JBoss environment.



4 The integration process

Figure 5: Iteration steps towards the TOSCA-MP Proof of Concept

In this chapter, the integration process that has been carried out to support the technical implementations of the system architecture, is summarised, see also Figure 5.

First, the task models that had been developed in WP4 as a fundament for the implementation of the business process models, were reanalysed and reworked to ensure their implementability by the PoC. This part of the process will be described in section 4.1.

Second, the technical implementation and publication of all PoC components as web services was coordinated, functional and technical tests were carried out and the technical integration of the services in the TOSCA-MP PoC was realised. This part of the process will be described in section 4.2.

Additional TOSCA-MP specific solutions that support the realisation of the PoC are described in section 4.3.

4.1 Realization of an executable Business Process Model

The practical implementation of the TOSCA-MP PoC should be able to execute (at runtime) the workflows that had been defined for the realization of the business goals. Already at the beginning of the project the partners had agreed to use a SOA based framework for the integration and realization of the PoC. A main requirement was that for the Metadata Production Management Framework (MPMF), responsible for the orchestration of services in the PoC, the business process should be made available by means of Business Process Model and Notation (BPMN 2.0).

The relevant scenarios for TOSCA-MP were derived as tasks (see D4.1) and formally represented as task models using the ConcurTaskTrees (CTT) metamodel; these formal task models were used to derive business processes and thus form the basis for the process model within the MPMF (see Figure 5). An automatic conversion was developed to transform the task models (exported in an XML representation of CTT) into business processes represented in BPMN 2.0 format. This conversion was implemented by means of Extensible Stylesheet Language Transformations (XSLT). The task model development, representation and conversion into BPMN2.0 were part of the activities in WP4.



Figure 6: Simplified representation of the Configurable Analysis Task Model

During a further analysis of the defined task models, focussing on the implementability in the PoC, it was found that various usage scenarios independent of each other use the same task in a different context. To avoid unnecessary duplication of tasks, and thus a duplication of process execution at runtime in the PoC, a common workflow – based on the so-called Configurable Analysis Task Model (CATM) – was defined, in coordination with WP4. The CATM combines all individual tasks of several usage scenarios into one single workflow. This ensures that a task is performed only once. Specifically the CATM was designed to support the project’s sports and news scenarios but due to its flexible configuration options is not limited to those applications: the flexible configuration in the MPMF allows tuning the set-up of the workflow according to the requirements of each specific usage scenario (see section 5.4.1).

A simplified representation of the CATM in Figure 6 (for a comprehensive overview please refer to Annex 9.2) shows various feature extraction and analysis sub processes that consist of multiple services (in blue), a human interaction task (red) and sub processes that are covered by a single service in the MPMF (yellow). Within the CATM some services depend on others; most importantly, the CATM includes a genre-dependent branching, in which one of three analysis sub processes in the process model is executed based on the results of the genre classification sub process. Each of these sub processes contains services that specifically support the further processing of essence having the appropriate genre.

For the CATM the BPMN 2.0 model was developed and improved to match the ongoing development of services, thus ensuring the implementability of the CATM by the PoC. Specifically the development of the feature analysis services, forming the core of the CATM functionality, was coordinated with WP2 to consolidate the available components for the system integration (see also D2.3, section 4 “From Business Goals to Metadata Extraction Services”). The resulting BPMN process (see Annex 9.2) supports the original formal task models defined in WP4, but makes sure that in a single “run” of the process any feature analysis service is not executed more than once.

4.2 Technical integration of services in the PoC

The tools (exposed as web services) that needed to be integrated were developed in various work packages:

automatic feature extraction and annotation services in WP2,

indexing and search in WP3,

services to access and manage the DRF in WP5 and

MPMF-related services in WP6.

These developments were the input to the technical integration work on the PoC (see Figure 5).

Based on the System Design the architecture for the TOSCA-MP PoC was iteratively implemented as part of the integration activities. All partners supported this process by updating the respective Service Description Sheets, detailing the descriptions of the planned Services (a.o. a functional abstract, the required in- & output data for the service and the planned Web service interface technology, see D6.3, Appendix A).

For the integration of the individual components into the TOSCA-MP POC web service technologies have been used. To streamline the web service integration amongst the partners, a set of “Collaborative Guidelines” was composed, listing the technical and functional requirements for web service



implementation and integration within the TOSCA-MP PoC3. As a main requirement, for each

component either a REST or a SOAP interface had to be provided (this choice was left to the courtesy of each partner) as these are the two most widespread interface technologies in the field of web services.

For each web service, the partners provided (and wherever required updated) a Service Interface Description document. These descriptions provided information on:

The web service functionality;

The protocols used;

The methods being made available via the web service interface;

Input parameters, return values and possible result codes.

This relevant detailed documentation allowed the system integrator to prepare for testing and integration of the web service-based components into the PoC, even in cases where the fully implemented component was not available yet: in such cases a respective “dummy” service was provided by the respective partner for testing, mimicking the actual service through its web interface.

Figure 7: SoapUI interface, showing Test Suite, Test Steps and results for the Concept Detection service

Before the actual integration of each component developed in WP2, WP3, WP5 and WP6, its web service interface was intensively tested by means of the open source testing software SoapUI

4 (see

Figure 7). Through appropriate testing projects in SoapUI the service accessibility, the validity of service requests and responses were verified (as defined in the Service Description Sheets and Interface Description Sheets), as well as the validity of the service output of each available service. The results were documented and bilaterally discussed allowing the resolution of errors and / or improvements. After successful testing, the services were subsequently integrated in the PoC by means of an appropriate “counterpart” within the MPMF – the so-called service client (see section 5.4.4). For additional testing two integration meetings were organized at IRT’s premises in Munich, to cooperatively solve problems and plan the next steps.

In total, over 35 services have been successfully tested and integrated in the PoC, as listed in Annex 9.3. That overview also defines the respective inputs and outputs (including Component Type) for each service, which the MPMF requires to prepare the DRF with the correct input and output types / parameters for each service method call.

3 The Collaborative Guidelines document is available from the project’s internal document management system: https://iis-bscw.joanneum.at/bscw/bscw.cgi/279559/TOSCAMP-IRT-Collaborative.Guidelines.for.Service.Implementation.v.1.1.docx

4 SoapUI is a free and open source platform with a graphical interface for the creation and execution of automated functional, compliance, and load tests for SOA and web services.

https://iis-bscw.joanneum.at/bscw/bscw.cgi/279559/TOSCAMP-IRT-Collaborative.Guidelines.for.Service.Implementation.v.1.1.docx

https://iis-bscw.joanneum.at/bscw/bscw.cgi/279559/TOSCAMP-IRT-Collaborative.Guidelines.for.Service.Implementation.v.1.1.docx



4.3 Additional TOSCA-MP-specific developments, supporting the

system integration

4.3.1 Handling of essence and metadata in a distributed environment

Figure 8: Handling of essence and metadata in the distributed PoC

In addition to using a SOA based platform, another prerequisite for the PoC was that large (essence and metadata) files were not to be transferred through the MPMF. Orchestration units (typically based on an Enterprise Service Bus) are not equipped to shuffle large amounts of bytes back and forth. Moreover, in the distributed environment of the TOSCA system architecture, having the centralized MPMF handle the essence files would mean, that these files would be transferred between DRF and MPMF and again between the MPMF and the appropriate service, thus unnecessarily duplicating the amount of data transfer (see also Figure 1).

To prevent large amounts of essence and/or metadata to flow through the MPMF, a URL-based communication concept was developed for the PoC to create and address resources in the Distributed Repository Framework (DRF) and communicate their locations between the MPMF and the appropriate services. During the initiation of a feature extraction (FET) service by the MPMF this concept foresees the following communication between MPMF / FET-services / DRF, see Figure 8:

The MPMF retrieves the location of the desired essence file in the DRF as a URL; also the MPMF creates one or more resources in the DRF to store the output of the FET service (metadata) after it has finished processing and obtains respective URL(s).

The MPMF calls the FET service through it web service interface and – amongst other parameters – hands over the URLs.

The FET service retrieves the essence from the DRF and starts processing.

The FET service stores its resulting output(s) (metadata) in the DRF and informs the MPMF that it has finished processing.

In a concluding step, the MPMF updates the resource file(s) stored by the FET service in the DRF.

This way, each FET-service itself is responsible for retrieving essence from and storing resulting metadata to the DRF; the MPMF obtains appropriate URLs from DRF and communicates them to FET-service. This meets the requirement that no essence or service outputs shall flow through MPMF; however, the MPMF handles the URLs to locate resources in the DRF as part of its responsibility for control and orchestration in the PoC.



This process, being generic to all FET service calls, is implemented in the MPMF as part of a generic service handler (see sections 5.4.3 and 5.4.4).

4.3.2 Adding Process Logic to the BPM

The CATM task model, created as Business Process Model and Notation (BPMN) file, had to be complemented to realize a SOA runtime environment with executable processes in the Metadata Production Management Framework (MPMF). The overall model was divided into one main process and relevant subsequent processes, see Annex 9.2. The sub-processes were designed as independent processes, decoupled from their particular parent process. By this approach each sub-process can also be executed separately, without the need of running the complete model.

In addition to the straight forward execution of tasks in parallel or serial, business relevant logic had to be added to the main process. The decision between the actual execution of the sports or news branch based on the outcome of the genre detection service (see Figure 6), was implemented by a complex gateway including a script and java source code, which reads the actual result of the genre service from the DRF and based on that initiates the respective – genre dependent – sub-process.

For each process specific input and output parameters have been defined to carry the relevant information, e.g. essence identifier, through the process tree. To gain additional flexibility beside those process parameters a configuration module was implemented. This gives the user the opportunity to configure each task in each process separately after the process has been deployed to the process engine, see section 5.4.1.



5 Integrated Proof of Concept realisation

This chapter describes how the Component Layer and the Orchestration Layer in the TSD (see Figure 2) have been realised to establish the technical basis of the PoC. For components developed in WP2, WP3 and WP5, an overview is given, focussing on the relevant aspects with respect to integration in the PoC; for details the reader is kindly referred to deliverables of the respective WPs.

5.1 DRF

The various local repositories in the PoC, which can consist of different types of internal storage, are encapsulated by so called LocalModules. Each Local Module has a common REST web service interface for reading and writing resources like video or metadata files and to read, write and query metadata about the stored content resources (for a detailed description see D5.1).

A LocalModule provides the content of a local repository in form of files to all TOSCA-MP user applications, analysis services and the MPMF. The information about the data that are stored in a LocalModule are recorded by an additional central unit, the so-called MainModule. That means, each LocalModule is connected with the MainModule and informs it about all internal content changes. For that, the DRF includes an internal messaging system and a number of predefined messages for notification of local content changes, as well as for internal configuration, administration and lifecycle management.

The MainModule collects the content information from all connected local repositories based on a specific internal metadata model, and provides this information to all TOSCA-MP user applications, analysis services and the MPMF based on a REST web service interface. This interface has the same structure as the interface of a LocalModule. For a read request to any content file in the DRF, a TOSCA-MP user application or service can directly query the global interface of the MainModule. In this case, it will receive the data over a relink direct from the local interface of the LocalModule that contains the requested data. For a write request, a user application or service uses the interface of that LocalModule in which it plans to store the new asset or metadata file.

Based on the simple resource oriented structure of the implemented REST web service based interfaces, the DRF could be easily integrated in the PoC. The same applies to the access to the DRF configuration data.

Please refer to D5.1 (section 3.5) for detailed information on the DRF REST API.

5.2 Content Analysis services

Several content analysis services have been implemented and integrated into the PoC. A brief overview is given here; further details can be found in D2.3.

5.2.1 Visual analysis

Shot boundary detection (S6) (see D6.3, section 4.3.1.6): Extracts shot boundary time points, types and transition duration if appropriate

Key frame extraction (S3): Extracts key frames and low-level descriptors

Face detection (S44): Detects faces in video content

Near-duplicate detection (S4): Identifies video segments that are near duplicates of other segments

Concept detection (S14): Identifies semantic concepts in image and video data

Stripe image extraction (S8): Extracts temporal overview images from columns of each frame

Visual Activity (S9): Extracts a global visual activity measure

Quality Analysis services (S5, S21a, S21b): Automatic no-reference defect analysis of video material

Football related services:

o Player Detection (S19): Detects players on a football pitch

o Player Identification (S20): Identifies players on a football pitch



o Sport Camera View Classification service (S22): Provides a sport specific shot detection and classifies these shots into relevant camera view categories

o Highlight Detection (S17): Detects football-related highlights

Action/Event detection (S1): Identifies actions and events that are in a given content

The following service has been implemented but not integrated due to time constraints:

Logo detection (S18, cp. D2.2 and D2.3 section 6.3): Detects logos in football content

5.2.2 Audio/text analysis

Spoken Language Identification (S13): Identifies the language spoken an in audio stream

Automatic Speech Recognition (S11): generates computer transcriptions of a speech content in two modalities: manual (supervised) or automatic (unsupervised).

Audio Segmentation and Speaker Clustering (S10): Audio-based speaker detection and grouping

Machine Translation (S12): Automatic translation of sentences from a source to a target language

Named entity recognition (S32) (see D2.3, section 5.3): Tags named entities in speech data

5.2.3 Multimodal analysis

Genre classification (S29): Classifies the genre of audio visual content

Cross Modal Segmentation Service (S30) (see D2.2, section 6.1): Extracts and identifies stories in English spoken news broadcasts

The following services have been implemented but not integrated due to time constraints:

Event Detection for Sport Video (S48): Multimodal event detection for sport videos

Names and Faces Alignment (S47) : Automatic alignment of names and faces of persons in video content

5.2.4 Low-level feature extraction and utilities

FET Service (S46) – extraction of low-level and structural features from audio visual content required by the genre recogniser (S29)

SIFT (S23): Provides a scale invariant feature transform e.g. for object recognition or concept detection

For the integration of the individual components into the TOSCA-MP PoC web service interfaces have been realised for the respective analysis services. Service description sheets were composed and the testing and integration process was carried out according to the description in section 4.2. A description of all service interfaces is out of scope here; for further details please refer to the detailed service overview in Annex 9.3.

5.3 Additional services

This section gives an overview of those services that do not analyse AV content but that have relevant additional functionality in the PoC that appears to an end-user e.g. via one of the user interfaces (see also chapter 6).

5.3.1 Index Service / Search service

The purpose of the indexing service is to gather all the high- and low- level annotations extracted by the audio-visual analysis services and stored in the Distributed Repository Framework to create a semantic-based inverted index by fusing low- and high-level features and enhancing those annotations with knowledge from external ontologies (e.g. Linked Data ontologies). Besides other semantic mechanisms (e.g. semantic query expansion), such an inverted index is used by the search service to provide a retrieval engine in the TOSCA-MP project. The indexing service is orchestrated by the Metadata Production Management Framework as the final step in the CATM once all the analysis services have finalised; additionally, also manual invocation is allowed.



The indexing service is exposed in two flavours:

1) A simplified indexing API with two operations to start an indexing job and to check the status of the indexing process. It only receives a parameter - media item identifier in the DRF- and it returns a simple XML file.

2) An extended indexing API with a number of operations (including operations for stopping, restarting, abandoning a particular indexing job) and returning a more detailed XML file with information about the whole process (starting and ending time, internal step…).

The search service encapsulates a Semantic Search Engine and allows the retrieval of multimedia assets from the TOSCA-MP platform combining different underlying technology such as full-text search, semantic query module, probabilistic enhancement, synonym query expansion, hit highlighting, etc. Moreover, it applies different mechanisms from the Semantic Web and Natural Language Processing field to analyse and improve user queries by exploiting concepts and relationships within ontologies. The search engine is highly scalable, providing distributed search and index replication as well as navigation features.

The search service is exposed as a web service where all the different search parameters can be expressed as a JSON Search Criteria Object. This object is quite flexible and allows different level of complexity to express the query: a numbers of elements and arrays of elements with modifiers can be used to build a complex semantic query in the underlying search system. At this point, the search service is mainly exploited through the search user interface directly (see section 6 for more information) but also by the clustering view provided by VRT.

More technical details and information about the indexing and search services can be found in D3.3 and D3.4.

5.3.2 Benchmarking service

The benchmarking service implements the core part of the benchmarking workflow and contains the following steps:

Determine the differences between content analysis results and a reference (e.g., ground truth), resulting in a set of edits for each pair of result and reference

Perform one or more of the following

o Determine performance measures from the edits

o Train a model of the error dependencies and run error prediction or cost simulation

The components and the data flow within the benchmarking service are shown in Figure 9.

Result generation with different parameters as input to the benchmarking process is run from the Control & Configuration UI (see Section 6.1). The benchmarking service is part of the dedicated benchmarking process that is also invoked from the Control & Configuration UI.



Simulation

Benchmarking

Data analysis

Groundtruth/

reference

Groundtruth/

reference

ResultResult

Differencing Errors(Edits)

Errors(Edits)

Error measurement

Error prediction

Taskmodel

Taskmodel

Simulation Unitcosts

Unitcosts

Costestimate

Costestimate

Perfor-mance

estimate

Perfor-mance

estimate

Perfor-mance

measure

Perfor-mance

measure

Error modelling

ResultResult

Figure 9: Implementation of the benchmarking service

The benchmarking service is implemented as RESTful web service. The service receives benchmarking jobs containing the pointer to a reference metadata item on the DRF as well as to a set of result metadata items. The configuration of the service determines whether the models are updated, and whether error prediction and simulation are performed. The results are stored as and XML document on the DRF.

5.3.3 Export & import Service to / from Metadata Schema Dependent Repository Extension

An export (S53) and import (S52) service to and from the metadata schema dependent repository extension of the DRF have been integrated to be manually controlled by an additional web application (developed as part of the work in WP5). In the final TOSCA-MP PoC, the metadata import is integrated via a newly developed service (S51), which collects all required components for a given object-name from the DRF and imports them via the import service (S52) of the schema dependent repository. The service is triggered by the Control & Configuration GUI (see section 6).

5.3.4 Verify Quality Analysis service

Figure 10: Job queue of the verification service

The verify quality analysis service (S45) involves a human verification step in the CATM (see Annex 9.2). Jobs from the MPMF are queued in the job list component of the verification service (see Figure 10). The user can select a queued job and verify the detections by using the Quality Summary user interface described later in this section. After the quality annotations have been verified, they can be



uploaded to the destination URL as specified in the job description, by clicking the "Verification Finished" button of the job list component. When clicking the "Discard" button, the unmodified MPEG-7 file is uploaded to the destination URL.

The interface of the verify quality analysis service is exactly the same as for the quality analysis service (S5). The job description has the following format:

qualityMpeg7Uri=<uri to the original MPEG-7 file from the quality analysis

service S5>

kfMpeg7Uri=<uri to the MPEG-7 file of the keyframe extraction service S3>

kfMeUri=<uri to the file containing the keyframe images generated by S3>

siMpeg7Uri=<uri to the MPEG-7 file of the stripeimage extraction service S8>

siMeUri=<uri to the file containing the stripeimage files generated by S8>

outputMpeg7Uri=<uri where the final verified MPEG-7 file will be uploaded>

analysisProfileName=<instructions for the user, e.g. Fast Verification>

docId=<Object Name of DRF media item>

Figure 11: Quality Summary user interface used for verifying quality annotations

The interactive verification is done with the Quality Summary user interface shown in Figure 11. Before opening a job, the verify quality analysis service fetches the required data from the DRF and passes it to Quality Summary. The defects detected by the quality analysis service S5 are listed in the Defect Annotation component. Each defect can be approved or discarded, and new defects can be manually added. The resulting verified quality MPEG-7 document is then uploaded to the DRF.

5.3.5 Audio Extraction service

Some feature extraction services in TOSCA-MP need audio files for processing. To support this requirement a service for audio extraction was developed and integrated into the MPMF handling video



content which doesn't come with separate audio essence. The core of the service builds the FFmpeg5

transcoder, which demultiplexes the audio track from the audio visual essence and transcodes it to a mono wave file with a 16kHz sample rate. The service also handles the download of video files and the upload of the generated audio files from and to the DRF. The service was integrated into the MPMF via a FIMS interface

6, based on the FIMS transform media interface definition v.1.0.7.

5.3.6 Basket Sharing service

The basket sharing service allows users to create content baskets, put content in the baskets and share baskets between users. It consists of a simple data model of baskets, users and content as illustrated in Figure 12.

Furthermore, a REST interface has been provided to interact with the service. Among others, the main functionalities are creating and deleting baskets, listing, adding and removing users with which a basket is shared, and listing, adding and removing the set of content which is present in the basket. It also includes search functionalities to find all baskets of a particular user and to find all baskets in which a particular content item is present.

Figure 12: Shared content basket service data model

5.4 MPMF

This section describes the implementation of the framework for service orchestration and execution of media analysis processes, called Metadata Production Management Framework (MPMF). The framework takes the business process model as input for the orchestration within the PoC, and – with help of an additional configuration information – implements the CATM, allowing the execution of AV content analysis jobs and monitoring its progress.

Within the distributed media processing system, the MPMF fulfils the role of a mediator, interconnecting, controlling and orchestrating all other components via web service interfaces and supporting the communication and exchange of control- and metadata between them. Figure 13 shows the PoC system overview, showing the MPMF architecture in more detail.

5 http://www.ffmpeg.org/

6 http://www.fims.tv/



Figure 13: MPMF detailed architecture within the TOSCA-MP PoC

The Red Hat JBoss® Enterprise Application Platform provides a generic process engine (jBPM7) which

can be accessed for process control and monitoring through a standard web service interface. jBPM is a Business Process Management (BPM) suite based on a light-weight, extensible workflow engine written in Java that allows execution of business processes conforming to the BPMN 2.0 specification. Hence, all additional TOSCA-specific MPMF components have been developed in Java (JDK 6) as well.

Along with the Control & Configuration application (depicted as Control & Config GUI in Figure 13) the MPMF provides two main functionalities to an administrator / end user:

1. Configuration of a workflow (Control & Config GUI – Workflow Configuration) for flexible adjustment of all process and service parameters;

2. Process execution and monitoring (Control & Config GUI – jBPM5 Process Engine – TOSCA Integration Layer).

The Control & Configuration application gains access to the workflow configuration via a web service that was implemented by IRT. All workflow configurations are contained in an eXist XML database for reading, editing and saving.

Workflows (i.e. processes) can be executed via a JBoss out-of-the-box web service, giving access to the jBPM5 process engine as well as to a management console supporting process instance management, task lists and task form management, and reporting.

Within the TOSCA Integration Layer, the following components have been realized:

An adapted formal BPMN 2.0 representation of the CATM process to be executed by the MPMF;

A Generic Service Process, for each FET service handling the sequence of clients to be called;

Clients handling the communication with the DRF and the FET services;

For each integrated service an additional software module has been implemented to handle the service specific communication (not shown in Figure 13).

The following subsections provide more detailed information on the MPMF implementation.

5.4.1 Flexible workflow configuration

In addition to the JBoss extensions in the integration layer, the MPMF was equipped with an XML database (eXist 2.0

8) which provides the possibility to store workflow configurations. For each single

7 http://www.jboss.org/jbpm

8 http://exist-db.org/



service which is included in a business process leaf a workflow configuration specifies all parameters which are used during the process execution. An MPMF-internal service was developed to support the workflow configuration functionality, providing a web service interface for external access. Through this web service interface, the external Control & Configuration GUI can read, optionally modify and store configurations before starting a process (see section 6.1).

The workflow configuration is managed separately and only loaded at runtime of the process. Before the user starts the process, he can make the relevant settings for each corresponding task. When starting the process, the relevant configuration must be referenced. In addition to the user modifiable settings in the configuration, it is also possible to define which service is executed for a specific task in the given workflow. This way, it is possible to (de-) activate relevant services in a process to flexibly support multiple usage scenarios.

The workflow configuration is described in XML and stored in the XML database. The XML structure is defined by a corresponding XML schema definition. Each configuration consists of a unique identifier and a list of complex service objects, see Figure 14. A service object references the client interface to use, the endpoint where to reach the service and a list of the parameters for the in- & output of the actual service. Each service object can be set to inactive, in which case the process engine skips the service within the workflow.

Figure 14: Service object XML structure

5.4.2 BPMN implementation of the CATM

The process realising the CATM was directly modelled in the JBoss environment. This has been described in section 4.1 and Annex 9.2.

5.4.3 Generic service process

To fulfill a task in the context of a TOSCA workflow several pre-processing steps have to be done before actually invoking an analysis service. Therefore a generic service process with a modular concept has been developed to technically fulfill the functional requirement of each and every task. The generic service process handles the procedure described in section 4.3.1 and is based on four modules (see Figure 15):

Configuration: Loads the corresponding workflow configuration for the specific task, containing essence input/output types and service parameters



DRF Preparation:

o Retrieve the essence inputs from the DRF

o Prepare the essence outputs in the DRF

Service Invocation: Run the configured analysis service

DRF Update: Check the essence output of the service is available in the DRF

Figure 15: Generic service process within the MPMF

Since this process – used by any other high-level workflow in TOSCA – builds the interface or the gateway to all external components provided by WP2, it is essential for a running system and errors have to be detected quickly. In a second iteration of the development of the PoC, error handling and logging was added to the generic service process.

Error handling and logging

The standard behavior of the MPMF without error handling was to cancel the whole process in case an error occurred. To the user (administrator) an error description was only displayed as highly technical output in the BPMN console. Additionally the error was recorded in the server log, where it was accessible only for the administrators of the system.

To avoid these process breakups and to give users well defined and clear information about the error or breakup in the process, detailed error handling was added to the PoC implementation. The error handling considers following aspects:

o system architecture:

The TOSCA-MP PoC is a SOA based, distributed system. The error handling therefore must differentiate between errors which are caused by system parts that are not accessible by the MPMF (i.e. the services) and thus cannot be influenced by the MPMF, and errors that can be influenced by the MPMF.

o processes in the system:

The CATM process in the system combines all individual tasks of several usage scenarios into one single analysis workflow (see section 4.1). Therefore, the CATM process consists of several sub processes (which again may contain sub processes) and services. This makes the process very complex and in case of an error it is not always clear which sub process or service is responsible for it.

The error handling must therefore be able to register which sub process or service has thrown an error, so that the MPMF is capable of stopping just this sub process or service and continue with the execution of the process.

o types of errors with which the MPMF has to deal with:

There are two types of errors:

Non-critical errors recoverable errors

Such errors are the result of a failure to a business rule like “configuration is not available”. Other kinds of non-critical errors are e.g. errors caused by a system unavailability like “requested service is unavailable”. These must be handled by the MPMF.

Critical errors non-recoverable errors

Here it must be distinguished between:

1. Technical or development errors, like syntax and logic errors.

The technical or development errors on the MPMF side were fixed during the implementation and testing phase (see section 4.2).

2. Critical errors that occur at runtime (e.g. internal server errors): these must be handled by the MPMF.

http://dict.leo.org/#/search=postprocessing&searchLoc=0&resultOrder=basic&multiwordShowSingle=on



Figure 16: Generic service process including error handling

All error types are resolved by the MPMF by taking alternative paths in the process (through the error handler) and bringing the process back into a defined status (see Figure 16). Thus, instead of cancelling the whole process, only the appropriate sub-process or service is being “skipped”. This behaviour was integrated in the generic service process by means of a process error variable and the addition of a gateway after every module of the generic service process. After the execution of each module, in the gateway the process error variable is checked. If the error value is “clean”, the execution continues with the next module. If an error occurred, the execution is routed to the error path where the “Tosca Error Handler” specifies the error and generates the error message.

In addition to the error detection and resolution, all errors and process steps are being logged in the MPMF. This workflow logging was implemented specifically to give administrators a tool at hand to deal with the occurring errors but can also be used to review a passed through process at a later stage or as input to a benchmarking workflow.

For the log file composition, JBoss provides a basic logging function with the “LogEvent” class and its subsections. This class logs different events of the execution from different instances in one log file (see Figure 17). With this principle of logging the basic information is stored in one file, but the reader has a big problem of assigning the different events to the according originator. This log file structure does not support an administrator to quickly find solutions to problems that occur.

Figure 17: Log file example of the JBoss LogEvent



Figure 18: Log file object XML schema (see annex 9.4 for a sample log file)



To get a better structured log file, a new log file format for the MPMF was developed in TOSCA-MP. This structure maps the complete process into an XML structure, allowing an administrator to easily find the location of an error (e.g. in a component or during the handover between a sub-process and process or process and service) (see annex 9.4).

This log file is described in XML. The XML structure is defined by a corresponding XML schema definition (see Figure 18). Each log file consists of a unique identifier, general log file information, an error summary and one complex process object. This process object can be the cover for one single service or many complex sub-processes.

5.4.4 Generic service and DRF handler

The integration layer in the MPMF consists of all classes that were developed to extend the JBoss environment with the specific functionality for the MPMF. To support the correct execution of a formal process loaded onto the JBoss platform as BPMN, a generic service handler was implemented. For each leaf in the formal BPMN process model this handler takes care of the more detailed steps to be taken in the following sequence:

reading the parameters from the appropriate workflow configuration;

initializing the content/metadata repository;

calling the automatic annotation service and monitoring its execution; and

updating the content/metadata repository.

In each of these steps, the generic service handler accesses the appropriate software modules provided as Java classes to perform the following actions:

the communication with the content/metadata repository using the appropriate web service interface;

the communication with analysis services through the respective web service interface; and

handling of execution errors, either at the process level or internally in the MPMF for those errors that are thrown during the communication with the analysis and repository services.

The BPMN processes generated from the task models form a solid base for the concrete implementation of the scenarios to executable workflows. Nevertheless, some adaptations were required to meet technical requirements of the JBoss SOA platform, especially to handle decision logic at the start and at the end of process branches. Custom classes had to be developed to realize this process logic based on the workflow configuration parameters, e.g. to handle branching within the process dependent on the output of a service. In addition to that, routines for the abovementioned error handling were developed that were not considered from the perspective of task models. These routines are relevant for the technical implementation and are required to guarantee an error-free operational MPMF.



6 Graphical User Interfaces and applications

This chapter describes the Graphical User Interfaces / Applications which have been implemented, thus realizing the respective layer in the TSD (see Figure 2). A first version for some of these has already been reported as part of D6.5.

6.1 Control & Configuration

6.1.1 Functionality

The TOSCA-MP Control & Configuration GUI enables the user to check the status of DRF modules, to select content and a configuration for an analysis process, to start analysis processes and monitor them.

The GUI is structured by a tabbed view. Each tab is dedicated to one main use case. The TOSCA-MP Control & Configuration GUI is a web-application, thus only a web browser is needed on the client side. Deployment to a server is realised by means of a WAR (Web application ARchive) file.

DRF status

A table shows the name, the type, the URL and the status of the DRF main module and all connected local modules, see Figure 19.

Figure 19: The DRF Tab

Processing items

The processing tab, shown in Figure 20, provides functionality for selecting workflows, their parameters and for selecting the content objects to be processed (the Control & Configuration GUI communicates with the MPMF to control and monitor the process execution). Selected objects can also be grouped to content baskets:

Select one available analysis process



Select one configuration. Only those configurations which belong to the selected process will be suggested to the user.

Find content objects by name on all accessible DRF modules

Present the search result within a table

Allow selecting several content objects at once and start the selected analysis process with them

Download, prepare and present results of one content object within the AVDP viewer (see section 6.2)

Invoke the interactive manual annotation tool for one content object (see section 6.3).

Allow to define the name of a new content basket and select several content objects to assign them to the new content basket.

Figure 20: Tab „Process Start Items“ – Select process, configuration and start an analysis process for each selected content item

Processing using content baskets

The processing tab for content baskets, shown in Figure 21, provides similar functionality as the processing tab, but works on content baskets:


Select one configuration. Only those configurations which belong to the selected process will be suggested to the user.

Find content baskets by name


Allow to select several content baskets at once and start the selected analysis process with them. For each item in a selected content basket one analysis process will be started

Allow to define the name of a new content basket and select several content objects to assign them to the new content basket.



Figure 21: Tab „Process Start Content baskets“ – Select process, configuration and start an analysis process for each item in each selected content basket

Benchmarking

The Control & Configuration GUI allows performing benchmarking, and implements the user interface related steps of the benchmarking workflow. The entire workflow consists of the following steps, see Figure 22:


Select one configuration. Only those configurations which belong to the selected process will be suggested.

Define parameter options which should be used for benchmarking

Find content baskets by name


Allow to select several content baskets at once and start the selected benchmarking process with them. For each item and each parameter option one analysis process will be started.

Run the benchmarking service on the generated results and the ground truth

Based on the results, select parameters and create/update a workflow configuration with these parameters

Figure 23 provides an overview of this workflow. Preparing and invoking jobs using the MPMF, invoking benchmarking jobs (executing the benchmarking service described in Section 5.3.2) and displaying the benchmarking results is implemented in the Control & Configuration UI. Job execution is done by the MPMF.

The data sets used for benchmarking are defined as content baskets. The result generation and the benchmarking service are then invoked for content baskets, enabling to easily keep track of the content items being used.



Figure 22: Tab „Benchmarking“: Allows to set up and start analysis processes for selected content baskets and the execution of a benchmarking process

Control Application

Groundtruth/

reference

Groundtruth/

reference

ResultResult

Costestimate

Costestimate

Perfor-mance

estimate

Perfor-mance

estimate

Perfor-mance

measure

Perfor-mance

measure

Parameter selection/update

WFConfig

WFConfig

ResultResult

Result generation

Analysis job

Benchmarking Service

Figure 23: Benchmarking workflow



Process monitoring

The process monitoring tab, shown in Figure 24, retrieves information from the MPMF on current processes (through the jBPM Process Engine Web service interface) and provides following functionality:

Present all running processes in a tree-view

Show graph of a selected running process and highlight the service which is currently being processed

Allow to delete a running process

Display the last started or deleted process

Figure 24: Tab „Process Monitoring“: Shows the status of all running processes

6.1.2 Related services

The Control & Configuration GUI invokes job execution at the MPMF, thus running any set of services orchestrated by the MPMF. This includes running the benchmarking service as part of a dedicated benchmarking workflow. For controlling and monitoring processes, the Control & Configuration GUI uses the jBPM Process Engine Web service interface; for retrieving and modifying workflow configurations, the workflow configuration service (S43) is used, see also Figure 13.

Two services are executed directly (without intervention by the MPMF): Import/export from metadata scheme dependent repositories (by means of S51) is used to fetch and prepare data from the DRF before manual annotation and update DRF resources after changes done by the user. The basket sharing service (S34) is used to create content baskets and use content baskets for running analysis or benchmarking jobs.

6.1.3 Role in the overall workflow

The Control & Configuration GUI serves as the central user interface for running and monitoring analysis processes and accessing analysis results. It is the front end for the user to send jobs to the MPMF, selecting the workflow, configuration and content. For the content selection, the DRF search or content baskets can be used. The Control & Configuration GUI also implements benchmarking functionality, making use of the MPMF to run jobs and the benchmarking service.



6.2 AVDP Viewer

6.2.1 Functionality

The AVDP viewer presents timeline-based metadata of one MPEG-7 document. The viewer provides playback and navigation in a video, synchronising a video player with different metadata views. Interaction is possible on all visualisation components. The AVDP viewer is implemented in HTML5, and thus only needs a current web browser as client. It is deployed as part of the Control & Configuration GUI. Figure 25 provides an overview of the AVDP viewer, which contains the following views:

Segment-based views for shot boundaries and quality analysis events

A video player

A table of quality analysis events

Key frames and stripe images

Graph views for quality measurements

Speech-to-text and machine translation results, with one utterance per line, and colours coding the speaker

Figure 25: User interface of the AVDP viewer

The slider on the left allows zooming the temporal range of the timelines below the video player. This allows seeing more detail of the respective views, and the key frame view displays more key frames when zoomed in. Figure 26 shows and example of this.

The table view can also be used to navigate to the events listed there, and to use the player for looping around the event segment (see Figure 27). If needed, the player can be detached into a separate window, e.g. to display it full screen on a second screen.



Figure 26: AVDP viewer, with some views zoomed in

Figure 27: Navigation using the events in the table


The AVDP viewer is not linked to a single service, but makes use of the results of the following metadata extraction services, that store their results in the DRF as MPEG-7 AVDP files:

S3 Key frame extraction

S5 Quality Analysis

S6 Shot Boundary Detection

S7 Specific content types

S8 Stripe image extraction

S9 Visual Activity

S11 Automatic Speech Recognition

S12 Machine Translation

S17 Highlight Detection

S22 Sport Camera View Classification


The AVDP viewer is used at the end of the analysis workflow in order to check and review the results from different automatic content analysis tools.



6.3 Interactive manual annotation

6.3.1 Functionality

The interactive manual semantic annotation tool presents two different views for visualizing as well as editing results from the automated feature extraction process, as it is implemented in the PoC by the CATM.

Figure 28 depicts the first view. It realizes the visualization of the shots in the video, the results of automatic temporal video segmentation and face detection in form of thumbnails. All types of metadata can be clustered by dragging and dropping of the thumbnails. In addition, incorrectly detected faces can be removed by dragging them in the bin and additional faces can be detected by marking areas in the video thumbnail.

Figure 28: User Interface 1 of manual annotation tool

The second view visualizes the automatically generated concept and near duplicate results for a selected shot (see Figure 29). The user can manually add and remove concepts and near duplicates.



Figure 29: User Interface 2 of manual annotation tool


For the PoC, output of services S3, S4, S14 and S44 has been combined in the Manual Annotation tool. The Manual Annotation interface has been integrated into the Control & Configuration GUI. From that user interface the services described in section 5.3.3 (S51, S52, S53) are called to prepare the metadata for a selected video asset (i.e. the output of the respective services mentioned before) for the annotation. The Manual Annotation view of the selected video can then subsequently be accessed through the Control & Configuration GUI.


The Manual Annotation tool is used to visualize, verify and edit metadata generated automatically by various automatic feature extraction tools. It can be used for example in a video archiving process which is supported by automatic metadata extraction services to allow the archivist to verify the results and correct or complement them if necessary.

6.4 Search

6.4.1 Functionality

The Search User interface provides different visual approaches to browse content by means of exploratory analysis in order to shield users from the complexity and heterogeneity of the underlying media assets and repositories. Such an exploratory-based search interface has been designed and implemented with the clear goal in mind of managing very large and disparate amounts of information, whilst enabling better and faster information access, sharing and discovery. Various configurable controls (e.g. the Ontology Relationship Viewer, Dynamic Facets) enable the iterative expansion of queries with the aim of visually and progressively refining the result set.

An exemplary screen shot of the resulting Semantic Multimedia Annotation and Search interface with some of its visual components is shown in Figure 30. It shows the results for the textual query “Kate Middleton”. The system identifies the entity “Catherine, the Duchesses of Cambridge”, list the results with that information in mind and supports the navigation process through graph-based structures (e.g. showing how different entities are connected in the domain). Thus, users can easily navigate the graph-based structure of ontologies in search of a concept or set of concepts to directly refine the result set by indicating whether concepts should be added or removed from the result set.



Figure 30: Semantic Multimedia Annotation and Search interface

Alternatively, a view in which the videos have been visually clustered together is available, as illustrated in Figure 31. It contains a set with an automatically or user defined number of clusters. For each of the clusters, a word cloud generated with available textual metadata, and a set of prototypical keyframe examples of content in the cluster is shown.

In order to navigate through the results, the user can click on a particular cluster, upon which the search results in that cluster will be subdivided into a new set of sub clusters. In this way, the user can visually navigate through the search results. This view on the results is meant to be complementary to the list view presented earlier.

For more information on the cluster-based search result view, please refer to TOSCA-MP deliverable D3.4.



Figure 31: Visual clustering of the search results


The Search interface exploits the Search service outlined in section 5.3.1. Every time the user queries the system, the interface calls the Search web service and retrieves and shows the list of media asset results. Next, any user interaction applying filters (e.g. length or provider of the video), including or excluding concepts into the query from the Ontology Relation Viewer or Ontology Browser, etc. invokes the Search service modifying the internal JSON Search Criteria Object with more complex parameters. All these information is available due to previous indexing process through the Indexing Service, which creates a semantic inverted index, explained in Section 5.3.1.


In terms of scenarios (and business goals) defined in the project, the use of the search feature can be found across a number of them since search has become a commodity and a fundamental step in daily activities. Such search service and interface is deeply important in business goals such as the retrieval of very recent material (BG1.1) and historical archive material (BG1.2), search of archived and deep-archived material (BG1.2_S1, BG2.1_S1), but also as part of more elaborated processes - e.g. assisting users to create a summary of a football season (BG8_S2).

More information about the search user interface can be found in D3.3 and D3.4.

6.5 Football match summary

6.5.1 Functionality

To demonstrate the usefulness of the sports related metadata extraction services a dedicated graphical user interface was developed which supports the assisted production of sports events. The user interface provides tools for navigating videos showing football matches (a screenshot is shown in Figure



32). It is meant to demonstrate a summary generation workflow for e.g. a sports show. The software provides a video player and additionally a timeline widget (a) that shows different types of automatically detected events (such as goals, fouls, goal attempts). It allows for quick navigation and selection of scenes that might be of interest for the generation of a summary clip. Additionally, different camera view classes can be visualized in the timeline. This information can be used as an additional indicator for editors to identify interesting scenes, because shorter shots and multiple changes of the camera view class often appear in those scenes.

Figure 32: User interface of sports events visualization

Furthermore, player detection results (visualized by bounding boxes in the video), and field registration results (b) can be displayed within the video. If player identification results are available, player names are displayed below the player. That can help editors to distinguish players, while the field registration visualization can e.g. help an editor to better estimate player positions and distances to the goal.


For this tool, mainly the football related services are used to extract the metadata, i.e. the highlight detection service (S17), the player detection service (S19) and the camera view classification service (S22). The metadata produced from these services is combined into a single AVDP document, which is passed to the sports visualization tool and displayed in the GUI.


This tool is typically applied at the end of the analysis workflow in case the “sports” branch has been selected and thus the services mentioned in 6.5.2 have been invoked. In terms of business goals and scenarios, it mainly supports the assisted production of sports events. While both scenarios (BG8_S1 and BG8_S2) are relevant for this tool, it is especially helpful for the creation of a summary of a football match (BG8_S1).

b

a



7 Conclusions

The integrated Proof of Concept (PoC) as it was planned by TOSCA-MP has been successfully delivered. The PoC consists of a Distributed Repository Framework (DRF), a large collection of audio visual (AV) content analysis services, the Metadata Production Management Framework (MPMF), several Graphical User Interfaces (GUIs) and some additional services which do not analyse AV content but have relevant additional functionality in the PoC that appears to an end-user.

The integration of the components developed in WP2, WP3, WP5 and WP6 was an iterative process, which was accompanied with supportive documentation provided by all partners (e.g. Service Description Sheets, Interface Descriptions and Implementation Guidelines). The system architecture (see also D6.3) follows the principles of Service-oriented Architecture (SOA), as it was planned from the start of the project; hence, relevant components have been integrated by means of web services. The formal descriptions of all components (as web services) and their interfaces facilitated extensive and iterative testing before the actual integration of the web service-based components into the PoC.

Over 35 services have been integrated in the TOSCA-MP PoC. Most of them are being orchestrated by the MPMF, which implements the framework for orchestration and execution of media analysis processes, based on the task models and business processes that were designed for the realisation of the project’s scenarios. In some cases, services are directly being accessed by another component or from a GUI.

The MPMF has been realised based on the Red Hat JBoss® Enterprise Application Platform, and contains a multitude of TOSCA-MP specific solutions for a flexible workflow configuration, pre-processing steps (before the actual invocation of analysis services), process logic and an extensive error handling and logging. For the handling of essence and metadata in the distributed PoC, a solution based on URL-exchange between MPMF and analysis services was implemented, thus preventing large (essence and metadata) files from being transferred through the MPMF.

Two main processes have been implemented in the MPMF: one for benchmarking and one for a flexible analysis of AV content. The latter is based on the so-called Configurable Analysis Task Model as a means of combining the individual formal task models of several usage scenarios in a single analysis workflow, to avoid unnecessary duplication at runtime.

Following GUIs were implemented to provide the PoC functionality to the end user:

Control & Configuration: this GUI enables the user to check the status of DRF modules, to select content and a configuration for an analysis process, to start analysis processes and monitor them.

AVDP viewer: this GUI presents timeline-based metadata of one MPEG-7 document.

Interactive manual annotation: this GUI allows visualizing as well as editing results from the automated feature extraction process.

Search: the search GUI provides different visual approaches to browse content by means of exploratory analysis.

Football match summary: this GUI supports the assisted production of sports events, by navigation of content from football matches, allowing an editor to create a summary for e.g. a sports show.

The functionality the GUIs provide to the end user, in combination with the underlying technical architecture of the PoC (MPMF, DRF, services) realises the project’s scenarios and business goals (see also D2.3).

The integrated PoC is a fully functional SOA based prototype which has already been used on multiple occasions for demonstration and dissemination purposes (e.g. during the project’s field trials). The project partners aim at having it available for a year after the project’s finalisation, to facilitate further dissemination of the results.



8 Glossary

Terms used within the TOSCA-MP project, sorted alphabetically

AMWA Advanced Media Workflow Association

DoW Description of Work

DRF Distributed Repository Framework

FIMS Framework for Interoperable Media Services

MPMF Metadata Production Framework

REST Representational State Transfer

SOA Service Oriented Architecture

SOAP Simple Object Access Protocol

WADL Web Application Description Language

WP Work Package

WSDL Web Service Description Language

Partner Acronyms

DTO Technicolor, DE

EBU European Broadcasting Union, CH

FBK Fondazione Bruno Kessler, IT

HHI Heinrich Hertz Institut, Fraunhofer Gesellschaft zur Förderung der Angewandten Forschung e.V., DE

IRT Institut für Rundfunktechnik GmbH, DE

K.U.Leuven Katholieke Universiteit Leuven, BE

JRS JOANNEUM RESEARCH Forschungsgesellschaft mbH, AT

PLY Playence S.L., ES

RAI Radiotelevisione Italiana S.p.a., IT

VRT De Vlaamse Radio en Televisieomroeporganisatie NV, BE

Acknowledgement: The research leading to these results has received funding from the European Union's Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 287532.



9 Annexes



9.1 Logical System Design – comprehensive version



9.2 Configurable Analysis Task Model – comprehensive version



9.3 Overview of the services within the TOSCA-MP PoC



2 3= component integrated; other values: component available at partner but not integrated

3 A '+' sign separates multiple (simultaneous) input- or outputtypes and their respectively possible MIME types



9.4 Sample XML view of the MPMF log file

Log file with expanded basic structure



Log file with partly expanded “SubProcess”-Node BasicFeatures

Final Report on Integration - Tosca-mp€¦ · Final Report on Integration Deliverable D6.6...

Documents

Transcript of Final Report on Integration - Tosca-mp€¦ · Final Report on Integration Deliverable D6.6...