Tutorial Outline

1

www.d4science.eu

Tutorial Outline

30’ •From Content Management Systems to VREs

50’ •Creating a VRE

80 •Using a VRE

20’ •Conclusions

From Content Management Systems to VREs: requirements, challenges, and opportunities

Pasquale [email protected]

RCDL 200810 October 2007Dubna (Russia)

www.d4science.eu

3

www.d4science.eu

Session Outline

• Towards VREs A bit of history Delos DL Reference Model VREs, VOs, and e-Infrastructures D4Science Vision

• gCube System Characteristics Technological Complexity and Solution Architecture

• gCube Standards and Technology SOI, WSRF gCore

• gCube Enabling Services Features Opportunities

From Content Management Systems to VREs10 October 2008, Dubna (Russia)

4

www.d4science.eu

The evolution

Virtual Research Environments

many virtual organizations

Repository +Catalogue +Search service

few large institutions

1996consumer

Digital Library

few small institutions

2001 consumer

2006 consumer and resource provider

Digital Library Management System

many small institutions

2003 consumer and data provider


5

www.d4science.eu

The DL Universe

DIENST

ADEPT

NSDL

NDLTD

DSPACE

FEDORA TEL

DRIVER

BRICKS DILIGENT

ACM DL

ECHO

PERSEUS

OPENDLIB

EUROPEANA


6

www.d4science.eu

DELOS: Digital Library definition

A (potentially virtual) organization that comprehensively collects, manages, and preserves for the long term rich digital content and offers to its user communities specialized functionality on that content, of measurable quality, and according to prescribed policies

* DELOS Reference Model for Digital Libraries


7

www.d4science.eu

DL Reference Model: the 7 Domains

The Reference Model is founded on 6+1 domains:

Resource – captures generic characteristics (super-domain)

Content – information available

User – actors interacting with system

Functionality –operations supported

Policy – rules and conditions governing operation

Quality – qualitative & quantitative characterisations of system

Architecture –physical software (and hardware) constituents concretely realising the DL* DELOS Reference Model for Digital Libraries


8

www.d4science.eu

DL Reference Model: the 7 Domains [cont.]

* DELOS Reference Model for Digital LibrariesFrom Content Management Systems to VREs10 October 2008, Dubna (Russia)

9

www.d4science.eu

Reference Frameworks


10

www.d4science.eu

Virtual Research Environments (VRE)

• Distributed frameworks for carrying out cooperative activities like “in silico experiments”, data analysis and processing, production of new knowledge using specialized tools

• Largely based on retrieval and access of always updated knowledge from diverse heterogeneous content sources

• Produce knowledge that is preserved and made available for other usages inside and outside the VRE


11

www.d4science.eu

Virtual Research Environments: characteristics

Highly dynamic, created and dismissed on-demand

Based on specialised tools which support the generation of new knowledge

M26

0

0,2

0,4

0,6

0,8

1

1,2

Info

rmai

on S

ervi

ce

Brok

er &

Mat

chm

aker Ke

eper

DVO

S

VDL

Gen

erat

or

Cont

ent

Man

agem

ent

Wra

ppe

r &

Mon

itor

Cont

ent

Secu

rity

Met

adat

a Br

oker

Anno

tati

on

Met

adat

a M

anag

emen

t

Dat

a Fu

sion CS

DS

Pers

onal

izat

ion

Ind

ex S

ervi

ce

Sear

ch S

ervi

ce

Feat

ure

Extr

acti

on S

ervi

ce

Proc

ess

Des

ign

& V

erifi

cati

on

Proc

ess

Exec

utio

n &

Rel

iabi

lity

Proc

ess

Opt

imiz

atio

n

Arte

Por

tal

ImpE

Ct

Port

al

PrototypeAvailableBuild

Operating on new information objects


12

www.d4science.eu

Virtual Research Environments: the vision

• A Virtual Research Environment (VRE) provides a framework of applications, services and data sources dynamically identified to support the underlying processes of research/collaboration/cooperation.

The purpose of a VRE is to help researchers* belonging to Virtual Organization by managing the increasingly complex range of tasks involved in carrying out their activities.

*Researcher has to be considered in the large, i.e. end-user, decision-makers, resource and data providers, etc.


13

www.d4science.eu

Virtual Organization (VO)

• A Virtual Organization (VO) models sets of users and resources belonging to a e-Infrastructure. It defines clearly and carefully

what is shared, who is allowed to share, and the conditions under which sharing occurs, usually

based on an authentication and authorization policies.

VOs may have a limited lifetime and they are dynamically created to satisfy transient needs of the constituent potentially heterogeneous actual Organizations.


14

www.d4science.eu

e-Infrastructure

• An infrastructure is the basic physical and organizational structures and facilities (roads, power supplies, ..) needed for the operation of a society or enterprise

• An e-Infrastructure provides support for effective consumption of shared resources:

hardware-bound resources (i.e. networks, storage, instruments, and computational resources),

system-level software resources (i.e. basic middleware services), and application-level software resources (i.e. data sources and services).

These e-Infrastructures offer mechanisms that concurrently exploit networks, grids and data in a seamless fashion, and will thus enable scientific communities to operate within a coherent model, regardless of the location of their research facilities.


15

www.d4science.eu

D4Science vision

D4Science visioncalls for the realization of scientific e-Infrastructures that will remove technical concerns from the minds of scientists, hide all related complexities from their perception, and enable users to focus on their science and collaborate on common research challenges

gCube isa framework to manage distributed e-infrastructures where it is possible to define, host, and maintain dynamic virtual environments capable to satisfy the collaboration needs of distributed Virtual Organizations (VOs)


16

www.d4science.eu

gCube System


17

www.d4science.eu

gCube Characteristics

A Software framework

to support ON-DEMAND virtual collaborations* among remote parties cost-effective, secure, dynamic, both short and long lived overcome ad-hoc systems alike

to make discoverable and accessible computing, storage, data and service resources

to promote and/or contribute to data and service integration

* Research EnvironmentFrom Content Management Systems to VREs10 October 2008, Dubna (Russia)

18

www.d4science.eu

gCube Technological Complexity

Software framework

needs a ‘middleware’ (typically distributed)

is open by definition new resource types and/or new resource instances can be de/registered at any

time

is powerful if it supports application scope the portion of the infrastructure in which a resource exists the portion of the infrastructure in which a resource can act, operate, or has

power or control

is powerful if it supports sharing scope (controlled resource sharing) machines, storage, data and services resources


19

www.d4science.eu

gCube Technological Solution

• By relying on gLite, gCube is an e-Infrastructure enabling system to share

computing, storage, data and service resources → g3

• gCube system allows collaborations in eScience strongly content-oriented, potentially data and processing intensive

within the sharing scope of Virtual Organizations (VOs) broader and longer lived may stretch across the whole infrastructure or else over significant subsets thereof

take place in Virtual Research Environments (VREs) scope interactively created, managed, defined, and used:

system administrators, application designers, researchers typically short to medium lifespan


20

www.d4science.eu

gCube Architecture

gCube framework is composed by 4 main subsystems:

Enabling Services definition and runtime management of VREs

Information Organization Services storage, management, description, and annotation of information in a VRE

Information Retrieval Services retrieval of information in a VRE

Presentation Services VRE users interface with system and application services


21

www.d4science.eu

gCube Standards and Technology


22

www.d4science.eu

Is a Virtualised IT Infrastructure which

1. exposes a catalog of services instead of running service instances,

2. supports Workflow definition and execution, and

3. includes infrastructure resources such as compute, storage, and networking hardware and software (middleware) to support the running of services.

Is a Virtualised IT Infrastructure which

1. exposes a catalog of WS instead of running service instances,

2. supports SOA Application, and

3. includes infrastructure resources such as compute, storage, and networking hardware and software (middleware) to support the running of services.

Service Oriented Infrastructure

gCube provides a production quality software framework to enable scientific e-Infrastructures empowered by a collaborative environment

SOI


23

www.d4science.eu

gCube & Standards for communication

• Java WSCore, Apache Axis, and GridForum specifications (and implementation if any):

WS-Notification, WS-Addressing, WS-Security

WSRFWS-ResourceProperties (WSRF-RP)WS-ResourceLifetime (WSRF-RL)WS-BaseFaults (WSRF-BF)WS-ServiceGroup (WSRF-SG)

WS-DAI, WS-DAIR, WS-DAIX


24

www.d4science.eu

Web Service Resource Framework (WSRF)

Web Services

Lifetime Properties Service

GroupsNotifi-cation

ErrorHandling

Unified way to model and interact with stateful web services

Logic

WS

Logic

Statefull WS

State


25

www.d4science.eu

Web Service Resource Framework (WSRF)

Lifetime (WS-ResourceLifetime) Factory based dynamic creation of

services Instances are created with a limited

lifetime Prevent services from consuming

resource indefinitely (“Garbage Collection”)

Properties (WS-ResourceProperties) Defines type and values of a resource

state

Web Services

Lifetime Properties Service

GroupsNotifi-cation

ErrorHandling

Service groups (WS-ServiceGroups) Describes an interface for operating on

collections of WS-Resources E.g. to distribute an action to a set of

services

Notification (WS-Notification) Notification about state changes Applies traditional publish/subscribe

paradigm

Error Handling (WS-BaseFaults) Defines base handling of communication

errorsFrom Content Management Systems to VREs10 October 2008, Dubna (Russia)

26

www.d4science.eu

WSRF: Advantages and Disadvantages

Advantages Clear Description of Resources and Interfaces Dynamic sharing of resources On-demand services exploitation Cross-organizations trusted environment Widely accepted Web Service standards

Disadvantages Reference implementations are still in development Several complementing specifications are in development Complex middleware requires maintenance and

administration overheadFrom Content Management Systems to VREs10 October 2008, Dubna (Russia)

27

www.d4science.eu

Towards the usability: gCore

To overcome the complexities of the design and implementation of SOI compliant services

an application framework for the consolidation / development of existing/new gCube services

the gCore Framework (gCF)

To meet the needs of system administrators, infrastructure managers, and resource providers

an easy to install self-contained service container for the participation to Service Oriented Infrastructures

the gCore Container (gCore)


28

www.d4science.eu

gCF facilities: partial overview

WS management

lifetime

activation updatere-deployement

failure

State

configuration

models

lifetime

scoping

publication

persistence

notification

Calls

scoping

faults

security

scoping

serviceport-types


29

www.d4science.eu

gCube & Standards for communication

• Mutual authentication based on GSI secure conversation (through delegation and renewal)

• Business Process Execution Language for Web Services (WS-BPEL)

• GridFTP and SRM support

• VOMS for users and groups management

• GWT and JSR168(JSR268 is coming)

https://quality.wiki.d4science.research-infrastructures.eu/quality/index.php/Standards




30

www.d4science.eu

gCube Enabling Services


31

www.d4science.eu


• VRE Management services definition of VREs the dynamic deployment of VRE resources across the infrastructure

• Software Repository service Storage and provision of deployable software components

• Information Service publication of resources profile discovery of VRE resources through xPath, XQuery real-time monitoring subscription/notification

Dynamic Virtual Organisation Support services robust and flexible security framework for managing VOs


32

www.d4science.eu

HWgHN

HWgHN

Service provision continuity

HWgHN

HW gHN

Balancing utilization with head room

Dynamic Load Balancing

WS

State

WS

State

CPU Usage30%

CPU Usage90%

Rapid deployment

Production

HWgHNSoftware

Repository

WS

…WS

State

WS

State

WS

State

WS

State


Soon available

Failure Recovery


33

www.d4science.eu

gCube goes beyond other management systems by adding Service Management capabilities

Provides solutions for system administrators to

eliminate manual deployment overheads, eliminate manual environment configuration overheads, ensure optimal placement of services within the infrastructure support user community services orchestration

Opens unique opportunities for outsourcing state-of-the-art service implementations



34

www.d4science.eu

Conclusions

• gCube infrastructures creates new opportunities to change the VRE development model used by distributed and dynamic organisations and communities

• gCube offers an application framework for the development of Stateful

Web Services (gCF) an easy to install self-contained service container (gCore) SOI middleware (Enabling services)

Rapid deployment Failure recovery Load balance (soon)


35

www.d4science.eu

http://www.gcube-system.org/ http://www.d4science.org/

http://www.d4science.org/

http://www.d4science.org/

36

www.d4science.eu

gCube & Standardization Bodies

• ISO: data representation (e.g. ISO3166 for countries, ISO4217 for currencies) and metadata (ISO19115 for GIS)

• OGF: Standards related to Architecture (e.g. OGSA), Data (e.g. DAIS, GridFTP), Management (e.g. GLUE, Resources Usage), Applications (e.g. DRMAA), Compute (e.g. JSDL)

• OAI: Resources Exposure/Harvesting (OAI-PMH) Resources Exchange (OAI-ORE)

• OASIS: Standards related to stateful web services (e.g. WSRF), process management (BPEL), remote user interfaces (WSRP), A&A (SAML / XACML)

• W3C: All the standards related to Web Architecture (e.g., URI/URL, HTTP), Service Oriented Architectures (e.g. SOAP, WSDL, WS-Addressing) and data representation and manipulation( e.g. XML*)

• Others: Classification systems (e.g. ISSCAAP, ISSCFV, ISSCFG), features representation (e.g. GML for GIS), metadata (e.g. AgMES for Agricultural, SDMX for Statistics)


Tutorial Outline

Documents

Transcript of Tutorial Outline