Project@ a glance

Project@ a glance

Digital Libraries Powered by the Grid

ii

Start date: September 1, 2004.

Duration: 36 months.

Effort: 1077 Personmonths.

Cost: €9,546,561.

European Commission Contribution: €6,300,000.

Administrative and Financial Coordinator: ERCIM.

Technical and Scientific Coordinator: CNR-ISTI.

Laying the foundations for next generation of collaboration

and knowledge management environments: deploying

Digital Libraries on-demand over Grid-enabled

infrastructures.

DILIGENT aims to establish a standards-based test-bed which integrates state-of-the-art Digital Library and Grid technologies to support a wide range of research and industrial applications. With DILIGENT, members of Virtual Organisations can create on-demand, transient Digital Libraries based on shared computing, storage, content, and application resources. A Knowledge Management Environment based on DILIGENT guarantees secure, coordinated, dynamic, and cost-effective collaboration and sharing of knowledge.

Furthermore, DILIGENT aims to:

open up Grid technology to a broader range of research and industrial communities;

promote the diffusion of Digital Libraries technology through a cost-effective operational model;

promote cross-fertilization between the Digital Libraries and Grid domains which will foster synergies and advances in both areas.

ObjectivesObjectives

TechnologyTechnology

The DILIGENT infrastructure is comprised by a set of interacting services which provide:

support for the creation and operation of on-demand, transient Digital Libraries;

typical Digital Library functionality, such as search, annotation, personalisation, and content visualisation;

access to third-party application and information sources;

functionality to handle shared content and application resources.

Furthermore, DILIGENT can integrate pre-existing infrastructure resources (nodes, services, etc) supplied by collaborating parties with its own ones and also exploit those provided by the EGEE Grid infrastructure.

As a result, the DILIGENT infrastructure includes hosting nodes, content resources, and application resources. Its component services can use any gLite-compliant infrastructure to carry out computational and storage intensive tasks.

DILIGENT project emphasises on

technical development in order to

deliver a usable platform.

Design and development tasks

form most of the project's activities

and are linked to the various

components of the DILIGENT

platform. Testing and integration

activities glue the components

together into a robust, stable

system that can be deployed,

reused or extended at will.

As a proof of concept, DILIGENT

supports two user communities

operating within disjoint

application domains:

ImpECt, focusing on the

analysis, study and prevention

in environmental research

through the management of

services and information for

science-oriented applications

in Earth Observation;

ARTE, targeting management

of rich-media archives and

enabling sharing and

dissemination of information /

knowledge produced by

research in humanities, social

sciences and communication.

Finally, a number of tasks promote

the sustainability of the project

and the maximisation of its impact

to the constituent technologies.

The Enabling Grids for E-sciencE (EGEE) project is funded by the European Commission and aims to build on recent advances in grid technology and develop a service grid infrastructure which is available to scientists 24 hours-a-day

A DΙgital Library Infrastructureon Grid EΝabled Technology

Platform Overview

The DILIGENT architecture is distributed across a stack of logical

layers built on top of EGEE middleware to achieve modularity

and independence from hosting system.

The DILIGENT platform spreads over five service groups and three logical layers. The Collective Layer lays the foundations for the system by enhancing existing Grid collective services in order to support the complex service interactions required by the Digital Library Layer.

The DILIGENT platform spreads over five service groups and three logical layers. The Collective Layer lays the foundations for the system by enhancing existing Grid collective services in order to support the complex service interactions required by the Digital Library Layer.

Service layersService layers

Digital Library Creation and ManagementDigital Library Creation and Management

The Information Service supports the discovery and real-time monitoring of DILIGENT resources. Single services and entire Digital Libraries can thus tune resource usage and balance workload.

The Keeper Service acts as the ‘orchestrator‘ of the services which form a Digital Library whilst assuring the Quality of Service required by the Digital Library's definition criteria. To do so, it monitors the status of services and resources during the lifetime of the Digital Library and accordingly re-designs its topology at runtime.

The Broker and Matchmaker Service facilitates the efficient usage of the Grid infrastructure through an optimal distribution of services and resources across Grid nodes. The service exploits and extends underlying grid middleware capabilities so as to match DILIGENT hosted Digital Libraries needs.

The Virtual Organisation Support Service is dedicated to the management of Virtual Organizations. A Virtual Organisation (VO) is a dynamic pool of distributed resources shared by a dynamic set of users from one or more organizations in a trusted way. In DILIGENT, Digital Libraries make use of the VO mechanism to glue together users and resources in the trusted environment of the Digital Library.

The Virtual Digital Library Generator Service allows users/communities to create their own Digital Libraries. It permits users to specify a set of criteria which define the characteristics expected from the new Digital Library; then, it identifies the services and information sources which are required to satisfy the criteria.

The Information Service supports the discovery and real-time monitoring of DILIGENT resources. Single services and entire Digital Libraries can thus tune resource usage and balance workload.

The Keeper Service acts as the ‘orchestrator‘ of the services which form a Digital Library whilst assuring the Quality of Service required by the Digital Library's definition criteria. To do so, it monitors the status of services and resources during the lifetime of the Digital Library and accordingly re-designs its topology at runtime.

The Broker and Matchmaker Service facilitates the efficient usage of the Grid infrastructure through an optimal distribution of services and resources across Grid nodes. The service exploits and extends underlying grid middleware capabilities so as to match DILIGENT hosted Digital Libraries needs.

The Virtual Organisation Support Service is dedicated to the management of Virtual Organizations. A Virtual Organisation (VO) is a dynamic pool of distributed resources shared by a dynamic set of users from one or more organizations in a trusted way. In DILIGENT, Digital Libraries make use of the VO mechanism to glue together users and resources in the trusted environment of the Digital Library.

The Virtual Digital Library Generator Service allows users/communities to create their own Digital Libraries. It permits users to specify a set of criteria which define the characteristics expected from the new Digital Library; then, it identifies the services and information sources which are required to satisfy the criteria.

The Digital Library Layer is the domain-specific functional layer of DILIGENT, for it supports the storage, handling, and retrieval of multi-type and mixed-media content.

Finally, the Application-Specific Layer gathers general-purpose application tools as well as APIs and SDKs for third parties to migrate their data or functional components to the DILIGENT platform. Furthermore it builds a number of tools of common interest to domain-specific application developers.

The Digital Library Layer is the domain-specific functional layer of DILIGENT, for it supports the storage, handling, and retrieval of multi-type and mixed-media content.

Finally, the Application-Specific Layer gathers general-purpose application tools as well as APIs and SDKs for third parties to migrate their data or functional components to the DILIGENT platform. Furthermore it builds a number of tools of common interest to domain-specific application developers.

DILIGENT adopts gLite as the underlying Grid middleware.

gLite services in use are:Job Management (WMS/CE/WN) in: Feature Extraction Service; Content Security Service; Process Execution and Reliability Service.Data Management (SE/LFC/FTS) in: Content Management Service; Keeper Service.Information System (RGMA) in: Information Service (DIS).Security (VOMS) in: Dynamic VO Support Service.

DILIGENT will eventually join EGEE production infrastructure.

DILIGENT adopts gLite as the underlying Grid middleware.

gLite services in use are:Job Management (WMS/CE/WN) in: Feature Extraction Service; Content Security Service; Process Execution and Reliability Service.Data Management (SE/LFC/FTS) in: Content Management Service; Keeper Service.Information System (RGMA) in: Information Service (DIS).Security (VOMS) in: Dynamic VO Support Service.

DILIGENT will eventually join EGEE production infrastructure.

EGEE in DILIGENTEGEE in DILIGENT

is the next generation middleware for grid computing, born from the collaborative efforts of several academic and industrial research centres funded by the EU as part of the EGEE Project. gLite provides a framework for building grid applications tapping into the power of distributed computing and storage resources across the Internet.

is the next generation middleware for grid computing, born from the collaborative efforts of several academic and industrial research centres funded by the EU as part of the EGEE Project. gLite provides a framework for building grid applications tapping into the power of distributed computing and storage resources across the Internet.

GT4 components in use are:Java WS Core (as hosting environment for

all DILIGENT services);Aggregator Framework (in Information

service);Credential Management (in Dynamic VO

Support);WS AuthN and AuthZ (in Dynamic VO

Support).

GT4 components in use are:Java WS Core (as hosting environment for

all DILIGENT services);Aggregator Framework (in Information

service);Credential Management (in Dynamic VO

Support);WS AuthN and AuthZ (in Dynamic VO

Support).

A Digital Library Infrastructure on Grid Enabled TechnologyA Digital Library Infrastructure on Grid Enabled Technology

Architecture, Services, Applications and Middleware in DILIGENT

Created by G.Kakaletris @ di.UoA (2006)Created by G.Kakaletris @ di.UoA (2006)

Process ManagementProcess Management

The Process Design & Verification Service provides the user with graphical tools for the design and manipulation of process definitions. Furthermore, before a compound service (i.e. a workflow of service invocations) may be deployed, the provided verification functionality validates its compliance with certain rules, so that the process can be considered safe for execution.

The Process Execution & Reliability Service is responsible for all actions pertaining to the actual execution of compound services, i.e. for finding and allocating resources, starting, monitoring or aborting processes and handling process execution failures in accordance with transactional policies/guarantees.

Within the Process Optimisation Service, user and system processes are being optimised prior to their execution by the actual services under a rich, flexible cost-estimation policy which takes into consideration a multitude of parameters and values.

The Process Design & Verification Service provides the user with graphical tools for the design and manipulation of process definitions. Furthermore, before a compound service (i.e. a workflow of service invocations) may be deployed, the provided verification functionality validates its compliance with certain rules, so that the process can be considered safe for execution.

The Process Execution & Reliability Service is responsible for all actions pertaining to the actual execution of compound services, i.e. for finding and allocating resources, starting, monitoring or aborting processes and handling process execution failures in accordance with transactional policies/guarantees.

Within the Process Optimisation Service, user and system processes are being optimised prior to their execution by the actual services under a rich, flexible cost-estimation policy which takes into consideration a multitude of parameters and values.

Content and Metadata ManagementContent and Metadata Management

Indexing and SearchingIndexing and Searching

The Content Management Service supports transparent access to both DILIGENT and external rich-media or plain content. It handles the creation of new objects and updates of existing ones via various techniques such as polling and notifications, while providing support for URI maintenance, replication and partitioning.

The Metadata Management Service manages metadata for the description and enrichment of the digital objects handled by the Content Management service.

The Metadata Broker Service provides the mechanisms for achieving metadata interoperability.

The Content Security Service secures the storage and dissemination of content, safeguarding both Intellectual Property Rights and content integrity through the application of techniques such as signing, watermarking etc.

The Annotation Management Service offers integrated management of annotations for digital objects available on the DILIGENT platform, primarily multimedia content objects. Its main goal is to support collaboration activities between distinguished users of a DILIGENT Digital Library.

The Content Management Service supports transparent access to both DILIGENT and external rich-media or plain content. It handles the creation of new objects and updates of existing ones via various techniques such as polling and notifications, while providing support for URI maintenance, replication and partitioning.

The Metadata Management Service manages metadata for the description and enrichment of the digital objects handled by the Content Management service.

The Metadata Broker Service provides the mechanisms for achieving metadata interoperability.

The Content Security Service secures the storage and dissemination of content, safeguarding both Intellectual Property Rights and content integrity through the application of techniques such as signing, watermarking etc.

The Annotation Management Service offers integrated management of annotations for digital objects available on the DILIGENT platform, primarily multimedia content objects. Its main goal is to support collaboration activities between distinguished users of a DILIGENT Digital Library.

The Feature Extraction Service provides automated metadata generation for image, video, audio, and text content objects. The metadata can then be used by the Search Service in order to look for objects by content or similarity.

The Index Service assists the search service by enabling efficient content-based retrieval. It leverages state-of-the-art search engines (such as FAST’s Enterprise Search Platform) to deliver advanced text searching, and it extends search functionality to other media objects by indexing results from feature extraction as well as other metadata.

The Content Source Description and Selection Service optimises the distribution of content-based queries across search engines available on the DILIGENT platform. Using up-to-date descriptions of the content targeted by each engine, the service automatically selects the engines which are most likely to produce relevant results for a given query.

The Data Fusion Service supports the real-time distribution of content-based queries across search engines available on the DILIGENT platform. Using statistics about the engines and their per-query outputs, the service normalises and merges autonomously produced result rankings.

The Search Service offers the entry point to the functionality of the entire search engine. It is a complex service which orchestrates the operation of various DILIGENT services under the user's request for information. Searches over structured, semi-structured and unstructured data are supported through potentially complex processing and transformations.

Through the Personalisation Service, queries are pre-processed and customised with user-specific data so as to tailor the results to the specific needs and expectations of the user. Personalisation criteria include cost, accuracy, content sources, formatting, etc.

The Feature Extraction Service provides automated metadata generation for image, video, audio, and text content objects. The metadata can then be used by the Search Service in order to look for objects by content or similarity.

The Index Service assists the search service by enabling efficient content-based retrieval. It leverages state-of-the-art search engines (such as FAST’s Enterprise Search Platform) to deliver advanced text searching, and it extends search functionality to other media objects by indexing results from feature extraction as well as other metadata.

The Content Source Description and Selection Service optimises the distribution of content-based queries across search engines available on the DILIGENT platform. Using up-to-date descriptions of the content targeted by each engine, the service automatically selects the engines which are most likely to produce relevant results for a given query.

The Data Fusion Service supports the real-time distribution of content-based queries across search engines available on the DILIGENT platform. Using statistics about the engines and their per-query outputs, the service normalises and merges autonomously produced result rankings.

The Search Service offers the entry point to the functionality of the entire search engine. It is a complex service which orchestrates the operation of various DILIGENT services under the user's request for information. Searches over structured, semi-structured and unstructured data are supported through potentially complex processing and transformations.

Through the Personalisation Service, queries are pre-processed and customised with user-specific data so as to tailor the results to the specific needs and expectations of the user. Personalisation criteria include cost, accuracy, content sources, formatting, etc.

Application Specific Services

Application Specific Services

DILIGENT offers a rich user interface to support the administration and management of the infrastructure. It also offers end-user interface allowing the full exploitation of platform’s potential on the fields of information hosting and retrieval.

Interfaces rely on Portlet technology and can thus be plugged into any standards-compliant Portal engine. Ultimately, users and administrators manage and use the DILIGENT platform under the intuitive interface of a personalisable web portal.

Besides mechanisms to support end-user interaction with the system, DILIGENT provides also basic components to visualise the main content types handled in the two application scenarios which form part of the project’s initial workplan.

DILIGENT offers a rich user interface to support the administration and management of the infrastructure. It also offers end-user interface allowing the full exploitation of platform’s potential on the fields of information hosting and retrieval.

Interfaces rely on Portlet technology and can thus be plugged into any standards-compliant Portal engine. Ultimately, users and administrators manage and use the DILIGENT platform under the intuitive interface of a personalisable web portal.

Besides mechanisms to support end-user interaction with the system, DILIGENT provides also basic components to visualise the main content types handled in the two application scenarios which form part of the project’s initial workplan.

The Globus Toolkit is an open source software toolkit used for building Grid systems and applications. It is being developed by the Globus Alliance and several other parties all over the world.

The toolkit includes software for security, information infrastructure, resource management, data management, communication, fault detection, and portability. It is packaged as a set of components that can be used either independently or together to develop applications.

The Globus Toolkit is an open source software toolkit used for building Grid systems and applications. It is being developed by the Globus Alliance and several other parties all over the world.

The toolkit includes software for security, information infrastructure, resource management, data management, communication, fault detection, and portability. It is packaged as a set of components that can be used either independently or together to develop applications.

WSRF and WSN specifications in use are:

WS-ResourceProperties (WSRF-RP);WS-ResourceLifetime (WSRF-RL);WS-BaseFaults (WSRF-BF);WS-ServiceGroup (WSRF-SG);WS-BaseNotification (WSN);WS-Topics (WSN).

DILIGENT services are WSRF compliant.

WSRF and WSN specifications in use are:

WS-ResourceProperties (WSRF-RP);WS-ResourceLifetime (WSRF-RL);WS-BaseFaults (WSRF-BF);WS-ServiceGroup (WSRF-SG);WS-BaseNotification (WSN);WS-Topics (WSN).

DILIGENT services are WSRF compliant.

Globus Toolkit in DILIGENTGlobus Toolkit in DILIGENT

Features @ a glanceFeatures @ a glance

DILIGENT is partially funded by the European Union under the Sixth Framework Program, Information Society Technologies priority (2002-2006)

The European Union is established in accordance with the Treaty on European Union (Maastricht). There are currently 25 Member States of the Union. It is based on the European Communities and the member states cooperation in the fields of Common Foreign and Security Policy and Justice and Home Affairs. The five main institutions of the European Union are the European Parliament, the Council of Ministers, the European Commission, the Court of Justice and the Court of Auditors. (http://europa.eu.int/)

Disclaimer: This publication has been produced with the assistance of the European Union. The contents of this publication is the sole responsibility of DILIGENT consortium and can in no way be taken to reflect the views of the European Union. Material presented in here by no means binds the DILIGENT consortium to future technological decisions and products.

Images sources: ARTE archives, EU and ESA public image galleries. Styling: UoA, ENG. Production: UoA.

** Work teams of marked partners moved to other organisations inside the project.

On-demand Digital Library creation, hosting and management;

Security integration through virtual organisation and virtual community support;

Content security handling (access and watermarking policies);

Semantic content management over a gLite superimposed storage management layer;

Metadata management and indexing;

Annotation management and visualisation;

Dynamic Digital Library resources reallocation;

Shared resource usage optimisation;

Complex process visual design, verification, optimisation and execution;

Advanced feature extraction through pluggable components and services;

Text, image, sound, video and multimedia content processing;

Information visualisation;

Information retrieval out of structured, semi-structured and unstructured data;

Support for application-specific extensions;

Provision of pluggable management and access web components;

Service Oriented Architecture paradigm based design;

Built on WSRF compliant service elements;

EGEE gLite middleware compliancy;

Implementation level homogeneity.

European Research Consortium for Informatics and Mathematics – France

Istituto di Scienza e Tecnologie dell'Informazione "A. Faedo“– Italy

National & Kapodistrian University of Athens – Greece

Swiss Federal Institute of Technology, Zurich** – Switzerland

Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. – Germany

University of Health Informatics and Technology Tyrol ** – Austria

University of Basel - Switzerland

European Organization for Nuclear Research – Switzerland

Engineering Ingegneria Informatica SpA Rome – Italy

University of Strathclyde – United Kingdom

Fast Search & Transfer ASA – Norway

European Space Agency – Italy

Scuola Normale Superiore – Italy

4D SOFT Software Development Ltd. – Hungary

RAI Radio Televisione Italiana – Italy

@@

http://www.diligentproject.org/

mailto:info@diligentproject.org

Contact Person:Donatella Castellidonatella.castelli@isti.cnr.itIstituto di Scienza e Tecnologie dell'Informazione "A. Faedo" - CNRVia Moruzzi, 1 , 56124, Pisa - ItalyTel: +39 050 3152902Fax: +39 050 3153464

DILIGENT design and implementation is built around the vision to provide a robust stable system capable of supporting production level Digital Libraries.

In order to make this possible procedures followed by industrial / commercial software development lines have been adopted and internal homogeneity has become a primary rule since the beginning of project, leading to:

full adoption of the Service Oriented Architecture as dictated by the OGSA paradigm;

adoption of JAVA as the sole development language and of WS-Core as the WSRF enabling toolkit;

application of common design patterns for families of problems;

development of horizontal mechanisms to support common tasks throughout the system;

provision of common terminology;

compliance with commonly accepted coding and documentation guidelines;

consistency in reuse of technologies and external components.

Despite the internal homogeneity, the Service Oriented nature of the system allows its full or partial reuse and extensibility by a diversity of heterogeneous systems that fit the industrial interfacing standards adopted by DILIGENT.

The frameworkThe framework