D4Science and ETICS Building and Testing gCube and gCore Pedro Andrade CERN EGEE’08 Conference 25...

D4Science and ETICS

Building and Testing gCube and gCore

Pedro AndradeCERN

EGEE’08 Conference

25 September 2008Istanbul (Turkey)

www.d4science.eu

2

www.d4science.eu

D4Science

D4Science is a FP7 project establishing a knowledge infrastructure of data, services and processing resources through the provision of Virtual Research Environments

D4Science simplifies the discovery, access and production of existing and new data through a collaborative environment

D4Science continues and consolidates the work started in the DILIGENT project opening it to new communities

D4Science provides a production environment for ESA & FAO

D4Science and ETICS25 September 2008, Istanbul (Turkey)

3

www.d4science.eu

gCube

gCube is the software implementing the D4Science resources sharing vision via Virtual Research Environments

gCube software: Provides a multi-domain Service Oriented Infrastructure Gathers data, services and hardware as system resources Supports on-demand virtual collaborations Reduces the cost of managing research infrastructures Tailored for e-Science (content oriented, data intensive) Is complaint with consolidated and emerging standards


4

www.d4science.eu

gCube

gCube resources: Computing and storage VRE services

content management search and index annotations

Collections raw data content metadata

Processes to manage other resources


5

www.d4science.eu

gCore

gCore is a framework developed to reduce the complexity of designing and implementing gCube SOI complaint servicesgCore is composed by: gCore Framework: An application framework to consolidate

and develop existing/new gCube services gCore Container: A self-contained software container

distributed to facilitate the deployment of gCube services


6

www.d4science.eu

D4Science in ETICS

Two projects: org.gcube: 24 subsystems org.gcore: 2 subsystems

225 components: 160 software components

51 services, 84 libraries, and 25 portlets Including services samples, external libs, test libs (18 components)

65 distribution components 64 gCube Software Archives gCore Hosting Node

Around 500 configurations


7

www.d4science.eu

D4Science Release Cycle


etics component configurations

etics subsystem configurations

etics project configurations

etics-build

etics-test

8

www.d4science.eu

Build

The objective of the build activity is to ensure api-level compliance of components and the creation of components’ artifacts.

ETICS is exploited to: Model build scripts and manage build dependencies Execute on-demand remote builds (via WA and CLI) Execute nightly local builds for development configurations Produce distribution artifacts (via dedicated components) Feed the D4Science reporting application with xml raw

reports


9

www.d4science.eu

Deployment Testing

The objective of the deployment testing activity is to ensure that the gCube software archives: Can be remotely and automatically deployed Once deployed, that it starts and is activated correctly

ETICS is exploited to: Manage the executing the deployment tests Define the deployment test tool dependencies Generate deployment test reports


10

www.d4science.eu

Deployment Test


ETICS Server

Condor Node

Condor Node

Condor Node

Condor Node

Condor Node

Condor Node

Service 2 Service 1Service A

Service 2 Service 1Service A

11

www.d4science.eu

Static Analyzers

The objective of the adoption of static analyzers is to improve the quality of gCore and gCube software

ETICS is planned to be exploited to: Run jFindbugs and jDepend (under adoption)

Other analyzers (not exploited via ETICS): Check coverage of javadoc documentation (checkstyle) Monitor the content of code repository

e.g. pre-compiled artifacts, etc.. Certification of distribution components

check structure, correctness, etc..


12

www.d4science.eu

Current Releases


13

www.d4science.eu

Current Releases


14

www.d4science.eu

Improvements

Parametric tests on remote infrastructure Single remote test with multiple parameterized tests

Co-scheduling mechanism to be further exploited Condor nodes firewall limitations

Notification mechanism ETICS is designed to support distributed teams No notifications between different ETICS roles

components availability successful build/test execution


15

www.d4science.eu

Improvements

ETICS configurations contains many information Can an ETICS API be useful ?

Reports summary information Summary page with built, test, and plug-ins execution Organized by configuration

Plug-ins availability Info about existing ones? How to add external plug-ins?


16

www.d4science.eu

Conclusions

Solid exploitation: DILIGENT/D4Science using ETICS since the early days ETICS is the default tool in D4Science build an test tasks ETICS completely adopted by all D4Science members

Solid collaboration: D4Science and ETICS plans kept aligned Efficient support from ETICS


D4Science and ETICS Building and Testing gCube and gCore Pedro Andrade CERN EGEE’08 Conference 25...

Documents

Transcript of D4Science and ETICS Building and Testing gCube and gCore Pedro Andrade CERN EGEE’08 Conference 25...