DJRA1.6.1 – Integration Work Plan and Status Report

EUROPEAN MIDDLEWARE

INITIATIVE

DJRA1.6.1 – INTEGRATION WORK PLAN AND STATUS REPORT

EC DELIVERABLE: D5.6.1

Document identifier: EMI-DJRA1.6.1-1277592-Integration_Work_Plan_v1.0.doc

Date: 31/07/2010

Activity: JRA1

Lead Partner: JUELICH

Document status: Final

Document link: http://cdsweb.cern.ch/record/1277592?ln=en

Abstract: This deliverable contains the detailed work plan of the integration activities and objectives compliant with the overall EMI Technical Development Plan and aligned with the work-plans of other tasks within this work package. The plan is released early in the project life and updated every year including a status report on the achievements of the past 12 months compared to the planned objectives.

http://cdsweb.cern.ch/record/1277592?ln=en�

DJRA1.6.1 – INTEGRATION WORK PLAN AND STATUS REPORT Doc. Identifier: EMI-DJRA1.6.1-1277592-Integration_Work_Plan_v1.0.doc Date: 31/07/2010

INFSO-RI-261611 2010 © Members of EMI collaboration PUBLIC 2 / 41

Copyright notice:

Copyright (c) Members of the EMI Collaboration. 2010.

See http://www.eu-emi.eu/about/Partners/ for details on the copyright holders.

EMI (“European Middleware Initiative”) is a project partially funded by the European Commission. For more information on the project, its partners and contributors please see http://www.eu-emi.eu.

This document is released under the Open Access license. You are permitted to copy and distribute verbatim copies of this document containing this copyright notice, but modifying this document is not allowed. You are permitted to copy this document in whole or in part into other documents if you attach the following reference to the copied elements: "Copyright (C) 2010. Members of the EMI Collaboration. http://www.eu-emi.eu ".

The information contained in this document represents the views of EMI as of the date they are published. EMI does not guarantee that any information contained herein is error-free, or up to date.

EMI MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, BY PUBLISHING THIS DOCUMENT.



Delivery Slip

Name Partner / Activity Date Signature

From André Giesler JUELICH/JRA1 03/01/2011

Reviewed by Balazs Konya,

Massimo Sgaravatto,

Jozef Cernak

LU/NA1, INFN/SA1, UPJS/SA2

12/02/2011

Approved by PEB

Document Log

Issue Date Comment Author / Partner 0.1 28/11/2010 Structure and Table of Content Alberto Di Meglio/CERN

0.2 06/12/2010 Starting with content André Giesler

0.3 15/12/2010 Defining integration test plan André Giesler

0.4 20/12/2010 Draft version for showing up progress André Giesler

0.5 23/12/2010 Refined draft version André Giesler

1.0 08/02/2011 Revised final version André Giesler

Document Change Record

Issue Item Reason for Change



TABLE OF CONTENTS 1 INTRODUCTION ......................................................................................................................................... 6

1.1 PURPOSE .................................................................................................................................................. 6 1.2 DOCUMENT ORGANISATION ..................................................................................................................... 6 1.3 REFERENCES ............................................................................................................................................ 6 1.4 DOCUMENT AMENDMENT PROCEDURE ..................................................................................................... 8 1.5 TERMINOLOGY ......................................................................................................................................... 8

2 EXECUTIVE SUMMARY ......................................................................................................................... 10 3 STATUS REPORT ...................................................................................................................................... 11 4 OVERVIEW OF REQUIRED WORK ...................................................................................................... 13

4.1 EMI 1 TIMELINES .................................................................................................................................. 13 4.2 EMI INTEGRATION AND INTEROPERABILITY OBJECTIVES ...................................................................... 13

4.2.1 Cross all areas ........................................................................................................................... 13 4.2.2 Compute Area ............................................................................................................................ 14 4.2.3 Data Area ................................................................................................................................... 14 4.2.4 Infrastructure Area .................................................................................................................... 15 4.2.5 Security Area.............................................................................................................................. 15

4.3 OVERVIEW OF PRODUCT CHANGES ........................................................................................................ 15 4.3.1 Compute Area ............................................................................................................................ 15 4.3.2 Data Area ................................................................................................................................... 16

5 SOFTWARE INTEGRATION AND INTEROPERABILITY STRATEGIES ..................................... 18 5.1 SYSTEM INTEGRATION AND TESTING ..................................................................................................... 18

5.1.1 How does Integration Testing fit into the Software Development Life Cycle? .......................... 18 5.1.2 Prerequisites for Integration Testing ......................................................................................... 18 5.1.3 Integration Testing Steps ........................................................................................................... 19 5.1.4 What is an Integration Test Plan? ............................................................................................. 19 5.1.5 How to write an Integration Test Case? .................................................................................... 19

5.2 UNIT INTEGRATION AND TESTING .......................................................................................................... 20 5.3 INTEROPERABILITY TESTING .................................................................................................................. 21 5.4 STANDARD COMPLIANCE TESTING ........................................................................................................ 22

5.4.1 Computer Hardware Resource Utilization ................................................................................ 22 5.4.2 Access for Review ...................................................................................................................... 23

6 PLANS FOR PERFORMING DETAILED SOFTWARE INTEGRATION AND INTEROPERABILITY ACTIVITIES ...................................................................................................... 24 6.1 PROJECT PLANNING AND OVERSIGHT ........................................................................................ 24

6.1.1 Integration Plan ......................................................................................................................... 24 6.1.2 Interoperability Plan .................................................................................................................. 24 6.1.3 Standard Compliance Plan ........................................................................................................ 25

6.2 SOFTWARE DEVELOPMENT ENVIRONMENT ............................................................................. 25 6.2.1 Software Engineering Environment ........................................................................................... 25 6.2.2 Software Test Environment ........................................................................................................ 26 6.2.3 Software Integration Tasks Tracking ......................................................................................... 28

6.3 INTEGRATION, INTEROPERABILTY AND STANDARD COMPLIANCE TASKS .............................................. 28 6.3.1 ARC Computing Element (A-REX) ............................................................................................ 29 6.3.2 WMS ........................................................................................................................................... 29 6.3.3 CREAM ...................................................................................................................................... 30



6.3.4 UNICORE compute elements ..................................................................................................... 31 6.3.5 ARC* Client ............................................................................................................................... 31 6.3.6 UCC Client ................................................................................................................................ 32 6.3.7 DPM ........................................................................................................................................... 33 6.3.8 dCache ....................................................................................................................................... 34 6.3.9 StoRM ........................................................................................................................................ 35 6.3.10 UNICORE data .......................................................................................................................... 37

6.4 OTHER SOFTWARE DEVELOPMENT ACTIVITIES ..................................................................................... 37 6.4.1 Risk Management ....................................................................................................................... 37

7 SCHEDULES AND ACTIVITY NETWORK .......................................................................................... 39 8 CONCLUSIONS .......................................................................................................................................... 41



1 INTRODUCTION

1.1 PURPOSE This document describes the EMI Integration Plan for the first year of EMI. It is applicable to the development and test activities within JRA1 that lead to the availability of software products conformant to the EMI technical objectives for Year 1 and compliant with the EMI Software Quality Assurance Plan and related procedures. This document is a specialization of the general EMI Development and Testing Plan describing in details the integration and interoperability strategies [R10].

1.2 DOCUMENT ORGANISATION This document is organized as follows:

Chapter 1 - Introduction: this section, explaining the purpose, scope and organization of the document

Chapter 2 - Executive Summary: this section contains a high-level description of the document. It gives a summary of the most important points described in each main section.

Chapter 3 – Status Report: this section describes the actual outcome of the integration tasks in the previous development cycle. Since we are considering the first cycle, an overview of the state of the art is given.

Chapter 4 - Overview of Required Work: this section describes the overall goals and timelines of the EMI-1 development activities.

Chapter 5 - Software Integration and Interoperability Strategies: this section describes the processes, tools and constraints applicable to the EMI-1 integration activities.

Chapter 6 - Plans for Performing Detailed Software Integration and Interoperability Activities: this section describes the actual implementation of the general processes and the detailed tasks to be performed by each Product within the scope of the EMI-1 integration activities.

Chapter 7 – Schedules and Activity Network: this section provides a global overview of the integration activities schedule.

Chapter 8 – Development Organization and Resources: this section gives a description of the integration organization, the roles and responsibilities.

1.3 REFERENCES R1 EMI JRA1.7 Wiki Page

https://twiki.cern.ch/twiki/bin/view/EMI/EmiJra1T7Integration R2 EMI Testbed Wiki Page

https://twiki.cern.ch/twiki/bin/view/EMI/TestBed R3 EMI JRA1 Wiki Page

https://twiki.cern.ch/twiki/bin/view/EMI/JRA1

R4 Technical Development Plan https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDNA131



R5 EMI Certification and Test Plan https://twiki.cern.ch/twiki/pub/EMI/EmiSa2CertTestGuidelines/EMI_SA2_CertificationAndTesting_v_1_0.pdf

R6 Compute Area Work Plan and Status Report https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDJRA111

R7 Data Area Work Plan and Status Report https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDJRA121

R8 Security Area Work Plan and Status Report https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDJRA131

R9 Infrastructure Area Work Plan and Status Report https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDJRA141

R10 JRA1 Development and Test Plan https://twiki.cern.ch/twiki/pub/EMI/InternalDeliverableEmi1ReleaseDevPlans/2010-12-17_EMI_1_Development_Test_Plan-v0.4_MRi_.doc

R11 GLUE2 Specification http://www.ogf.org/documents/GFD.147.pdf

R12 ETICS Build and Test System https://etics.cern.ch/eticsPortal/#legBs

R13 Build Configuration and Integration Guidelines https://twiki.cern.ch/twiki/bin/view/EMI/EmiSa2ConfigurationIntegrationGuidelines

R14 ETICS Nightly Builds http://etics-repository.cern.ch/repository/reports/name/emi_R_0/-/reports/index.html

R15 JRA1 EMI -1 Technical Objectives https://twiki.cern.ch/twiki/bin/view/EMI/EmiJra1T1EMI1TechnicalObjectives

R16 JRA1 Wiki pages with GANTT charts https://twiki.cern.ch/twiki/bin/view/EMI/EmiJra1T1SystemWPProgressTrackingProcess

R17 GLUE Integration Testing Plan https://twiki.cern.ch/twiki/bin/view/EMI/EMIJRA1T1EMI1TestStrategyGLUEInformation

R18 SRM HTTPS Binding Test https://twiki.cern.ch/twiki/bin/view/EMI/EMIJRA1T1EMI1TestStrategySRMCompliance

R19 SES Access Protocols Test Plan https://twiki.cern.ch/twiki/bin/view/EMI/EMIJRA1T1EMI1TestStrategySEsAccessProtocols

R20 GSTAT http://www.gstat.org

R21 Total Integration Matrix https://twiki.cern.ch/twiki/pub/EMI/EmiJra1T7Integration/IntegrationMatrix.xlsx

R22 Integration Compute Matrix https://twiki.cern.ch/twiki/pub/EMI/EmiJra1T7Integration/IntegrationCompute.xlsx

R23 Integration Data Matrix https://twiki.cern.ch/twiki/pub/EMI/EmiJra1T7Integration/IntegrationData.xlsx

https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDJRA111�




https://twiki.cern.ch/twiki/pub/EMI/InternalDeliverableEmi1ReleaseDevPlans/2010-12-17_EMI_1_Development_Test_Plan-v0.4_MRi_.doc�

https://twiki.cern.ch/twiki/pub/EMI/InternalDeliverableEmi1ReleaseDevPlans/2010-12-17_EMI_1_Development_Test_Plan-v0.4_MRi_.doc�

https://twiki.cern.ch/twiki/bin/view/EMI/�



R24 Integration Infrastructure Matrix https://twiki.cern.ch/twiki/pub/EMI/EmiJra1T7Integration/IntegrationInfrastructure.xlsx

R25 Integration Security Matrix https://twiki.cern.ch/twiki/pub/EMI/EmiJra1T7Integration/IntegrationSecurity.xlsx

R26 Total Interoperability Matrix https://twiki.cern.ch/twiki/pub/EMI/EmiJra1T7Integration/InteropMatrix.xlsx

R27 Interoperability Compute Matrix https://twiki.cern.ch/twiki/pub/EMI/EmiJra1T7Integration/InteropCompute.xlsx

R28 Interoperability Data Matrix https://twiki.cern.ch/twiki/pub/EMI/EmiJra1T7Integration/InteropData.xlsx

R29 Interoperability Infrastructure Matrix https://twiki.cern.ch/twiki/pub/EMI/EmiJra1T7Integration/InteropInfrastructure.xlsx

R30 Interoperability Security Matrix https://twiki.cern.ch/twiki/pub/EMI/EmiJra1T7Integration/InteropSecurity.xlsx

R31 Project Technical Board https://twiki.cern.ch/twiki/bin/view/EMI/PTB

1.4 DOCUMENT AMENDMENT PROCEDURE This document can be amended by the EMI JRA1 Integration Task Leader further to any feedback from other teams or people. Minor changes, such as spelling corrections, content formatting or minor text re-organisation not affecting the content and meaning of the document can be applied by the JRA1 Integration Task Leader without peer review. Other changes must be submitted to peer review and to the EMI PTB for approval.

When the document is modified for any reason, its version number shall be incremented accordingly. The document version number shall follow the standard EMI conventions for document versioning. The document shall be maintained in the CERN CDS repository and be made accessible through the OpenAIRE portal.

1.5 TERMINOLOGY DPM Disk Pool Manager

GLUE Grid Laboratory Uniform Environment

JSDL Job Submission and Description Language

GIN Grid Interoperation Now

KPI Key Performance Indicator

OGF Open Grid Forum

PGI Production Grid Infrastructure

SAGA Simple API for Grid Applications

SDTP JRA1 Software Development Test Plan

SRM Storage Resource Manager

https://twiki.cern.ch/twiki/bin/view/EMI/PTB�



TSI Target System Interface

UAS UNICORE Atomic Services

WMS Workload Management System

WebDAV Web-based Distributed Authoring and Versioning

XNJS Enhanced Network Job Supervisor



2 EXECUTIVE SUMMARY The EMI 1 Integration Work Plan and Status Report contain the detailed work plan of the integration activities and objectives compliant with the overall EMI Technical Development Plan and aligned with the work-plans of other tasks within JRA1. This document is closely intermeshed with the EMI 1 Software Development Test Plan (SDTP) [R10] which targets a couple of guidelines concerning the development and testing process of components. Although the plan is released early in the project life it is updated every year including a status report on the achievements of the past 12 months compared to the planned objectives.

The PTB has defined a number of technical objectives to be addressed as part of the EMI-1 release in addition to standard changes and improvements planned as part of the continuous product improvement activities [R31]. These objectives as far as they include integration and testing activities are covered in this plan.

Chapter 3 outlines an overview about the state-of-the-art concerning integration activities. Additionally it is described how JRA1.7 has enabled a continuous status request concept concerning the integration and interoperability tasks of EMI product teams.

Chapter 4 lists the particular objectives and highlights all affected EMI components in this regard. The overall strategies to test integration and interoperability in EMI are described in chapter 5. Developers and software testers will find here an approach how JRA1.7 performs integration, unit, and standard compliance testing.

Chapter 6 provides a fine-grained, detailed plan in which manner the strategies are adapted to the identified objectives from chapter 4. The detailed project schedule and activity network is described in the Section 7.



3 STATUS REPORT The current EMI development cycle is called EMI 0. It can be considered as an ‘exercise release’ designed to understand how to apply the agreed procedures, how to find any problem about the used build tools, and in general to fine tune the EMI software engineering process before the first real EMI software release. The EMI 0 release is not designed for external users. The goal is instead to prepare a consistent, coherent repository of non-conflicting software packages by the end of 2010 without any specific commitment on functionality. For this reason, almost no developments were explicitly done for this release. So, EMI components, which are integrated in EMI 0, represent in principle the same development or production status as before EMI and can be considered as the latest state-of-the-art. In some cases product teams defined already release candidates for EMI 1 in their EMI 0 configurations.

Since project start we have worked on the integration and interoperability matrices and their continuous updates by inquiring the individual product teams and checking their working plans. Integration refers to a setup where one software component inherently uses another component to provide a dedicated functionality. Interoperability within this EMI task means that one component is interoperable with another if both integrating the same interface. Chapter 5 provides a more detailed introduction to integration and interoperability strategies in this task. So, a client who is compliant with that interface is able to communicate with both components without the need of any transformation in the protocol.

In the matrices we are tracking information about integration and interoperability relations between EMI components, interfaces, or external components in terms of their current integration status or release plans, supported platforms, and their integration in the ETICS- and EMI-testbed system as well. In particular, the status of an integration/interoperability relation can have the following conditions:

• Not-applicable (e.g. an EMI component is incompatible with SRM standard)

• Not-integrated (e.g. it is not planned to integrate a specific EMI component with SRM standard within the EMI project)

• Fully integrated (e.g. an EMI component integrates the GLUE 2.0 standard)

• emi-0, emi-1, emi-2, emi-x: Integration is planned in EMI-0, EMI-1, EMI-2, or EMI-X which means later in project but not yet specified precisely

The statuses for the interoperability evaluation have quite the same definitions:

• Not-applicable (e.g. an EMI component is not interoperable to with another EMI component)

• Not-interoperable (e.g. it is not planned to integrate a specific EMI component with SRM standard)

• Fully interoperable (e.g. an EMI component is fully interoperable with another component by using a defined standard like HTTPS)

• emi-0, emi-1, emi-2, emi-x: Interoperability is planned in EMI-0, EMI-1, EMI-2, or EMI-X which means later in project but not yet specified precisely

Figure 1 shows an excerpt of the integration matrix. This tool is based on MS Excel tables and it shows exactly where points of interest with reference to integration of components are, while the latest status can be tracked via the wiki page [R1]. While many fields of these matrices are empty, those that are filled indicate integration activity between.

As being the first report, we describe the major areas of work for component integrations and we expect to refine these descriptions as well as the availability of integration tests in updates of this



report, including important cross-area integrations (e.g. where compute hits security, etc.). This list is not complete and expected to be re-fined with the future updates of this report, including also the addition of emerging components (i.e. EMI Registry, messaging, etc.). However, we define major relevant areas of work that are important to ensure that already the EMI-1 release has relevant integration tests in place that in turn realize a high quality integrated set of release candidates. Note that the details of these working areas are encoded in the integration matrices on the wiki [R1].

Figure 1: Excerpt of the integration matrix

The following table provides references to the individual integration and interoperability matrices.

Type Area Reference

Integration Total (all areas covered) [R21]

Integration Compute [R22]

Integration Data [R23]

Integration Infrastructure [R24]

Integration Security [R25]

Interoperability Total (all areas covered) [R26]

Interoperability Compute [R27]

Interoperability Data [R28]

Interoperability Infrastructure [R29]

Interoperability Security [R30]



4 OVERVIEW OF REQUIRED WORK

4.1 EMI 1 TIMELINES This section describes the overall EMI 1 timelines within the context of the EMI Development Roadmap. The EMI Technical Development Plan [R4] of the EMI project provides defined technical objectives for the EMI development by representing a definition of the EMI software portfolio, and a comprehensive technical formulation of development objects coupled to components and delivery dates. In addition to this, this document provides low level, implementation-specific details of the technical objectives as well as a timeline of the development tasks within the context of the first EMI project year. So, this document covers all scheduled developments to be released in EMI 1 with a particular focus on standardization and integration plans.

The development and integration process starting from the state-of-the-art to EMI 1 includes goals for the four technical areas plus a specific area for cross-area concerns. Section [4.2.] will cover all technical objectives with a deadline for the EMI 1 release concerning real physical components and their system integration aspects. EMI 1 will be released in April 2011.

4.2 EMI INTEGRATION AND INTEROPERABILITY OBJECTIVES This section gives a brief overview of the EMI technical objectives in year 1 for each technical area with particular focus on their integration and interoperability aspects. Examples of such objectives are those involving different implementations of the same capability, the adoption of common interfaces, and the use of reusable or external components across sets of different products. In addition, this section covers the general requirements of integration with standard operating systems. Technical objectives that will not include any development and integration activity are not listed here. Compare chapter 3 of the SDTP [R10] to get a complete list of all physical and not physical technical objectives.

4.2.1 Cross all areas The following objectives are improvements for EMI components that are not covered by the Technical Development Plan for Year 1. All of the listed objectives are already in development or in beta productions status. If these component improvements will be really released in time for EMI 1 or in later minor releases towards EMI 1.1 and higher depends on the progress in the responsible product teams.

Providing WebDAV access

For EMI 1 dCache will provide WebDAV authentication via user/password.

Authenticated Web Interface

For EMI 1 dCache will provide an authenticated web interface to manage pools and other system components.

Re-balancing data on pools

For EMI 1 dCache will provide the functionality of Re-balancing data on pools after new pools are added or filled unbalanced.

Improved drain performance

For EMI 1 DPM will speed up the draining of data pools in case of hardware maintenance or decommissioning.

https://twiki.cern.ch/twiki/bin/edit/EMI/WebDav?topicparent=EMI.EmiJra1T1EMI1DevAndTestPlansDCache;nowysiwyg=1�



Disk server monitoring

In preparation of the EMI common task monitoring the DPM storage element will improve the internal monitoring and will provide more information to track down problems.

Disk server management

For EMI 1 DPM will provide mechanisms to steer the number of streams from individual pools to better manage resources.

Support for roles

DPM will enable that priorities can be given to particular roles.

High availability clusters

For EMI 1 StoRM will provide a cluster solution for high availability.

Log analyzer

For EMI 1 StoRM will provide an administrative tool to interrogate StoRM log files.

Replacement of update-crls with fetch-crl

For EMI 1 StoRM will provide an administrative tool to interrogate StoRM log files.

EES development

The integration of the Argus policies into the mapfile generator should be completed.

4.2.2 Compute Area Glue 2.0 support in job management services and client tools

Several EMI components in the compute area are going to integrate the GLUE 2.0 model for taking advantage of this information schema. It is prerequisite that the EMI computing elements and the systems that offer access to computing systems should expose pieces of information about computing resources in a consistent manner. The integration of GLUE will be performed on unit level of the affected compute components by implementing the schema as XML or LDAP in the appropriate units. The implementations will be integrated in the ETICS system. The interoperability between components using GLUE as an information model will be tested by standard compliance tests which have to be defined within an integration testing plan. Chapter 6 provides a more detailed approach of the GLUE testing plans concerning EMI components.

4.2.3 Data Area Publishing GLUE 2.0 information in storage components

All three major storage elements within EMI named as dCache, DPM, and StoRM will integrate several information provider components. The goal is that EMI storage elements and the systems that offer access to storage systems should expose pieces of information about storage resources in a consistent manner. To ensure that the integration of these components is seamless, we also need to explore and execute integration tests in this context.

Using https for the SRM protocol

The SRM (Storage Resource Manager) is a protocol for Storage Resource Management. SRM is used to ask a Mass Storage System to make a file ready for transfer, or to create space in a disk cache to which a file can be uploaded. A prototype implementation should be created for EMI 1 where SRM is

https://twiki.cern.ch/twiki/bin/view/EMI/StoRM�

https://twiki.cern.ch/twiki/bin/view/EMI/StoRM�



using https as transport protocol instead of httpg. This prototype development explores the use of the SRM protocol without requiring the proprietary GSI. We will perform integration tests as soon as the prototype is available in the ETICS system.

HTTP(s) support for all storage elements

In order to be more in-line with modern Web-based access methods it is intended to provide HTTPS support for all EMI storage elements. The integration of this will be verified by functionality tests defined in a detailed test plan.

Supporting "file://" access protocol

All storage elements should offer at least a prototype-level support for the standard "file://" access protocol. The integration of this will be verified by functionality tests.

File Catalogue Access from UNICORE

Traditional the UNICORE system was not enabled to use any form of file catalogues and thus it is required to augment this functionality in order to fully function in the larger EMI ecosystem.

4.2.4 Infrastructure Area No objectives identified for EMI 1 in this area.

4.2.5 Security Area No objectives identified for EMI 1 in this area.

4.3 OVERVIEW OF PRODUCT CHANGES This section gives a brief overview of the EMI Products affected by the defined Technical Objectives as described in the overall EMI Software Development and Test Plan [R10] focusing on their integration and interoperability aspects. The tables in the following subsections provide information about the concerned EMI components, the current development status, the deadline of the task, and the destination platforms. However, the platform column shows which platforms are currently supported by the affected component. It had not yet been decided which platforms have to be supported in EMI 1 at the time of composing this document.

4.3.1 Compute Area Support for Glue 2.0 in job management services and client tools

Several EMI components are going to integrate the GLUE model for taking advantage of that information model standard. The following table lists the EMI components planning to integrate GLUE2.0 for the EMI 1 release

Component Status Integration due Platforms

A-REX In development M12 RHEL5, Deb5, Scientific Linux

CREAM In development M12 Scientific Linux

UNICORE TSI / U. XNJS / U.UAS/ WSRFLite

Beta version available M12 All platforms supported by Java



WMS In development M12 Scientific Linux

Libarcclient/ Arc* Beta version available M12 All Linux system, partly on Mac and Solaris

UCC (OGSA-BES plugin)/ HILA

Beta version available M12 All platforms supported by Java

4.3.2 Data Area Publishing GLUE 2.0 information in storage components

All EMI storage elements will publish initial GLUE 2.0 storage information, and a storage client or library should be capable to consume that information. The following components are affected in the context of this JRA1 technical objective.


DPM Beta version available M12 Scientific Linux

dCache Beta version available M12 All platforms supported by Java

StoRM Beta version available M12 Scientific Linux

UAS-D Beta version available M12 All platforms supported by Java


The following component is affected in the context of this technical objective.


dCache (Client and Server) Beta version available M12 All platforms supported by Java


The following components are affected in the context of this JRA1 technical objective.


DPM Not addressed by developers yet or already developed

M12 Scientific Linux

dCache Production quality M12 All platforms supported by Java

StoRM Beta version available M12 Scientific Linux

All storage elements offering at least a prototype-level support for the "file://" access protocol

The following components are affected in the context of this JRA1 technical objective.




DPM Beta version available M12 Scientific Linux

dCache Beta version available M12 All platforms supported by Java

StoRM In development M12 Scientific Linux

File Catalogue Access from UNICORE

Component In development Integration due Platforms

UAS-D Beta version available M12 All platforms supported by Java



5 SOFTWARE INTEGRATION AND INTEROPERABILITY STRATEGIES The subsections of this section describe the principles of different integration and interoperability strategies covered by this plan.

5.1 SYSTEM INTEGRATION AND TESTING This section describes the scope and goals of the system integration activity and the adopted testing methods and tools. Integration refers to a setup where an EMI software component inherently uses another software component to provide a dedicated functionality. The way how an EMI component integrates another one varies regarding the use of common interfaces and schemas (i.e. information model via GLUE) or simply the usage of libraries. Based on that definition integration tests are functionality tests involving the interaction among different components, generally developed by different Product Teams and which may be deployed on different hardware instances or as modules on the same instance.

Integration testing is a logical extension of unit testing. In its simplest form, two units that have already been tested are combined into a component and the interface between them is tested. A component, in this sense, refers to an integrated aggregate of more than one unit. In a realistic scenario, many units are combined into components, which in turn could be aggregated into even larger parts of the system. The idea is to test combinations of pieces and eventually expand the process to test particular modules with those of other components. Eventually all the modules making up a process are tested together. Beyond that, if the component is composed of more than one process, they should be tested in pairs rather than all at once.

Integration testing identifies problems that occur when units are combined. By using a test plan that requires EMI to test each unit and ensure the viability of each before combining units. This method reduces the number of possibilities to a far simpler level of analysis.

5.1.1 How does Integration Testing fit into the Software Development Life Cycle? Even if a software component is successfully unit tested, in an enterprise n-tier distributed application it is of little or no value if the component cannot be successfully integrated with the rest of the application. Once unit tested components are delivered they are integrated together. These integrated components are tested to weed out errors and bugs caused due to the integration. This is a very important step in the Software Development Life Cycle. It is possible that different programmers developed different components, so it is not unusual that a lot of bugs emerge during the integration step. In most cases a dedicated testing team focuses on Integration Testing. JRA1.7 will coordinate that process.

5.1.2 Prerequisites for Integration Testing The integration of components in EMI is initialised by adding appropriate modules to the EMI ETICS system. The responsible product teams create then ETICS configurations representing a specific version of an integrated component. These configurations contain information about used dependencies as well as build and test functions. The remote builds and tests are executed periodically from the ETICS build system, so that issues on that level are already logged and registered by the ETICS error reporting system. Additionally, ETICS is using specific plug-ins for static code analysis providing an analysis about the quality of the integrated code.

Before JRA1.7 begins with Integration Testing it is important that all the components have been successfully unit tested. This implies that the component has been already ported to the EMI ETICS system which is executing the unit tests periodically [section 5.2]. Another prerequisite is that the responsible product team has already produced a Software Verification and Validation Plan according to the EMI Testing and Certification guidelines [R5].



JRA1.7 has already registered all relevant integration tasks for EMI-1 in the integration matrices. These integration relations can be identified in the ETICS system by analyzing the appropriate configurations in regard to the used internal or external dependencies under reserve that the integrated task is represented by a library, service or other dependency. For example, the AMGA component integrates the BDII GLUE model. This integration task is represented by the ‘emi.bdii.glue-schema’ dependency within the AMGA configuration ‘emi.amga.glite-AMGA_postgres’.

5.1.3 Integration Testing Steps We have created a defined procedure which should on the one hand formalize the testing process, and on the other hand should also provide a manual for developers, testers and product coordinators who are involved in the process. The EMI integration testing for major releases typically involves the following steps:

1. Create a Test Plan (JRA1.7 and PTs)

2. Create Test Cases and Test Data (JRA1.7 and PTs)

3. Agreement with EMI SA2.6 and affected PTs how to utilize hardware resources and access the testbed (JRA1.7, PTs, and SA2)

4. If applicable create scripts to run test cases (JRA1.7 and PTs)

5. Once the components have been integrated execute the test cases (JRA1.7 and PTs)

6. Fix bugs if any and re-test the code (PTs)

7. Repeat the test cycle until the components have been successfully integrated (JRA1.7 and PTs)

These steps will be performed in close collaboration between JRA1.7, who will coordinate the testing process, the affected product teams, and the EMI-testbed administrators. While step 3 is described in section 5.4 and steps 4 to 7 are self-explanatory, the first two steps need more detailed description.

5.1.4 What is an Integration Test Plan?

This plan describes recurring parameters in a typical integration test case.

• How the tests will be carried out. • The list of things to be tested • Roles and Responsibilities • Prerequisites to begin Testing • Test Environment • Assumptions • What to do after a test is successfully carried out. • What to do if test fails • Glossary

5.1.5 How to write an Integration Test Case? In short, a Test Case describes exactly how the test should be carried out. The Integration test cases specifically focus on the flow of data/information/control from one component to the other. So the Integration Test cases should typically focus on scenarios where one component is being called from another. Also the overall component functionality should be tested to make sure the software product works when the different components are brought together. The various Integration Test Cases are



concatenated together forming an Integration Test Suite. Each suite may have a particular focus, which is usually defined by the technical objective. In other words different Test Suites may be created for on EMI component to focus on different technical objectives it is involved. As mentioned before a dedicated Testing Team may be created to execute the Integration test cases. Since JRA1.7 is not familiar with technical details of all EMI components, a strong collaboration with the appropriate PTs is required here. The Integration Test Cases should be as detailed as possible. Below is listed a sample Test Case table as planned for the integration testing.

Test Case

ID

Test Case Description

Input Data

Expected Result

Actual Result

Pass/Fail

Remarks

Additionally the following information will also be captured: a) Test Suite Name b) Tested by c) Date d) Test Iteration (One or more iterations of Integration testing may be performed)

When performing functionality tests defined in an Integration Test Case, JRA1.7 will go back to the Software Verification and Validation Plan of the component which has to be created by the responsible product teams according to Certification and Testing Guidelines [R5]. This report should already contain details of the available functionality tests of the affected component. In particular, it is listed what should be tested and how. If possible JRA1.7 will make use of existing test cases, or rather adopt them to the requirements of the integration testing. JRA1.7 checks that every predefined test concerning the integration task is running properly. The results will be added to the Software Verification and Validation Report [R5]. Integration testing is only part of the entire software development process. The latter is described comprehensively in the SDTP [R3].

5.2 UNIT INTEGRATION AND TESTING This section describes the scope and goals of the unit integration activity and the adopted testing methods and tools. Unit tests are intended to test the correctness of individual units or a group of related units. Since the components integrated in EMI are implemented in different programming languages, also the definition of a unit can vary accordingly. However, in general the methods of unit testing are well standardized. Tools like CPPUnit, JUnit, and PyUnit are predominantly used in EMI software components.

A unit is the smallest testable part of a software entity. In this respect, JRA1.7 will identify unit test cases in components affected in the integration activity that were explicitly implemented to validate the functionality. In a given case that means for instance, if an EMI component of the compute area has implemented the GLUE 2.0 schema successfully, there should exist as well some unit test cases which are validating the correct execution of the implementation.

JRA1.7 expects that appropriate unit tests in regard to the integration activity will be made available by the product teams along with the releases of the developments. Since ETICS is used in EMI as the common building and testing tool, each PT should provide already by default unit tests in their



appropriate component configurations. The unit tests should be also declared within the Software Verification and Validation Plan of the component [R5]. If no unit tests can be identified, we will request the developers to provide or if they are not sufficient to modify these tests. However, since EMI demands by default high unit test code coverage according to the Certification and Test Guidelines [R5], it can be expected that adequate unit test cases will be provided by the responsible developers. JRA1.7 will add the unit tests concerning the technical objectives defined in the SDTP to the overall integration test plan [section 5.2] to keep related unit and integration tests in a single document.

5.3 INTEROPERABILITY TESTING This section describes the scope and goals of the interoperability activity and the adopted testing methods and tools. Interoperability within this EMI task can be considered as follows. As shown exemplarily in Figure 2 service component A is interoperable with service component B since both integrate the same standard interface. In other words, any client that is compliant with this Standard Service Interface is able to communicate with both services without the need of any transformations in the protocol. In this particular context we can speak of a native interoperability between both components.

In contrast to the aforementioned form of interoperability, there is another type of interoperability that refers to the client to service communication. As also shown in Figure 2, the client application is interoperable with service component A and B, since client and server implement the same standard interface. While the client application uses a client implementation of this interface, the server components uses the server-side implementation of the interface.

Figure 2: Illustration of various interoperable components realized by adopting the same standard

interface on the server-side and on the client-side.



In principle, we will perform functional tests to check the interoperability between EMI components. So, JRA1.7 will create standardized tests which should be able to run on each EMI component which is implemented to process the used standard interface. For instance, we will create standardized jobs which will be submitted to all components in the compute area that are implementing a specific interface. In this respect the EMI testbed will be used predominantly as soon as the software products are available there. For EMI-1 all interoperability tests can be categorized as compliance checkers. The following section outlines how the technical objectives concerning interoperability are tested in a standard compliance testing approach.

5.4 STANDARD COMPLIANCE TESTING This section describes the scope and goals of the standard compliance activity and the adopted testing methods and tools. Since the project start we have already worked on the interoperability matrices and their continuous updates. In parallel, we have also started within this sub-task and identified a couple of initial standard compliance checkers together with JRA1.8 that make it possible to validate several interoperability works within the EMI components.

During the course of the project we foresee new standard compliance checkers to be created where necessary, for example, when specifications of the Production Grid Infrastructure (PGI) set of specifications/profiles will be available. JRA1.7 will closely work together with the developers in the product teams that require the existence of standard compliance checkers in order to ensure a high quality of their components. In this context, the interoperability matrix will point to relevant working areas and will enable to track progress towards the use of standard compliance checks. SRM Standard Compliance Checks This sub-task will also work together with the product teams in the data area that adopt the SRM interface. There is an SRMv2 compliance test suite available that is of major relevance for the EMI data component stack. The relevant component in context is dCache that already performs tests with this compliance test suite on a regular basis. Here it makes sense to broaden the usage of this test suite to other SRM-compliant systems such as DPM and StoRM. IPV6 Compliance Checks This sub-task will work together with those product teams of components that will take advantage of the emerging IPv6 technology. There is an IPv6 compliance test suite available and relevant for those components. The relevant component in context is currently the Logging and Bookkeeping service that already performs tests with this compliance test suite on a regular basis. Here it makes also sense to broaden the usage of this test suite to other EMI components that will make use of IPv6 during the course of the project. LDAP and GLUE Compliance Checks Early work exists within the EMI project to check the compliance to LDAP and GLUE in context. This sub-task will closely work together with those product teams that use LDAP. The relevant component in context is currently the set of BDII systems that are based on LDAP and take advantage of the GLUE information model standard. It will be explored how the set of components can be extended that take advantage of these tests as they are also extended towards the test of GLUE2 compliance. The particular tool is GStat [R20] that provides suitable validation probes.

5.4.1 Computer Hardware Resource Utilization This paragraph describes the approach to be followed for allocating computer hardware resources and monitoring their utilization. EMI has created an integration testbed with a set of allocated hardware resources to enable continuous software development and testing. EMI Integration Testbed indicates the whole set of geographically distributed infrastructural (hardware, networking, software) and operational (procedures, tools, documentation, communication tool, support handling etc.) resources made available for integration testing purposes. An overview of the EMI integration testbed and how



to use it in particular is described in [R3]. EMI task SA2.4 is in charge of the EMI integration testbed [R5].

JRA1.7 needs to perform functionality tests to check if an integrated task is really in place and working properly. These tests can only be done manually by using an appropriate installation of the affected EMI components and having authenticated/authorized access to them. Depending on the component, it may include interaction with other components developed by the same or a different product team. The right place to perform functional tests is usually the EMI integration testbed [R2], which provides operational resources made explicitly available for the continuous integration testing process of EMI software products. JRA1.7 will request access to the needed resources in the testbed to check if the integration task is working properly according to the defined Integration Test Plan [section 5.1.4] and thus tested successfully for a major EMI release. Finally, if system integration tasks cannot be tested by accessing installed resources within the EMI testbed for whatever reasons, JRA1.7 will try to install the identified components itself for further analysis.

JRA1.7 will hand out a special form of the integration matrix to EMI SA2.4 that contains all identified integration tasks of EMI components concerning EMI-1. The matrix provides information about which groups of components are affected in an integration task, which hardware resources need to be allocated, and which operation systems are in the scope.

5.4.2 Access for Review This paragraph describes the approach to be followed for providing project members access to developer work for review of software products and activities. Refer to ETICS public results, nightly builds results, test reports, etc.

Initially, project members can use the ETICS build and test system [R12] as a central point to get access to the available software components of EMI. Users can simply browse through the component tree of the ETICS web portal (or alternatively by using the ETICS command line client) to find the appropriate components and to examine the current configuration of the component. A quick check in the ETICS workspace will give an overview of used dependencies and if unit tests are enabled in remote builds and tests. According to the Build Configuration and Integration Guidelines [R13] the formatted names of the ETICS configurations characterize their state. For instance, the letter ‘R’ describes that the configuration is a release version of the component.

All component configurations associated with an EMI-release candidate are executed periodically in nightly builds. For EMI-0 this automatic build is already activated and are summarized in an overall ETICS report [R14] which is freely accessible for EMI project members. This constant updated web-document provides detailed build and test reports of all adopted components.

Additionally, JRA1 provides comprehensive information at the TWIKI pages. A good starting point about ongoing developments for EMI releases, the technical objectives, and testing strategies is the JRA1 EMI -1 Technical Objectives page [R15]. Finally, the development activities are tracked via GANTT charts that are publicly available on a quarterly basis in the JRA1 Wiki pages [R16]. This enables the EMT, PTB, and PEB the access to the status and results of the developments.



6 PLANS FOR PERFORMING DETAILED SOFTWARE INTEGRATION AND INTEROPERABILITY ACTIVITIES

Several parts of this section refer to the corresponding sections in the EMI-1 Development and Test Plan.

6.1 PROJECT PLANNING AND OVERSIGHT This section describes the types and content of the development and test plans of each EMI component with a particular focus on the EMI-1 release. For each product the available plans are included by reference or marked as missing if still in development.

6.1.1 Integration Plan Each affected Product Team must develop and record plans for conducting the integration activities required by this document and by other software-related requirements in the project. This planning must be consistent with system-level planning and shall include all applicable items in the SDTP. In particular it must describe the system and unit integration test cases and test plans.

These integration plans should be part of the Software Development Plans which are already requested from each Product team as defined in the JRA1 SDTP [R3]. In these plans, each of the PTs provides a detailed description of their system and integration test cases and test plans of the affected EMI components. JRA1 are actually part of the work plans for each technical area within EMI and can be found via the following references.

Integration Activity Technical Area Relevant Components of PTs Plan

Support for Glue 2.0 Compute A-REX, CREAM, U. TSI, U. XNJS, UAS-C, WSRFlite, WMS, libarcclient, arc*, UCC

[R17]

Publishing GLUE 2.0 Data DPM, StoRM, dCache, UAS-D [R17]


Data dCache (client and server) [R18]


Data DPM, dCache, StoRM [R19]

Support for the "file://" access protocol

Data DPM, dCache, StoRM [R19]

File catalogue access for UNICORE

Data UAS-D In development

From SDTP: Note that with respect to EMI-1, the infrastructure area has no implementations that directly address EMI technical objectives as part of this plan, but plans of these PTs can be obtained via [R8, R9] and will be included in the next update of the development plan for a EMI minor release or the next major release.

6.1.2 Interoperability Plan Each affected Product Team must develop and record plans for conducting the interoperability activities required by this document and by other software-related requirements in the project. This planning must be consistent with system-level planning and shall include all applicable items in the SDTP. In particular it must describe the interoperability test cases and test plans.

No interoperability tasks are planned for EMI 1.



6.1.3 Standard Compliance Plan Each affected Product Team must develop and record plans for conducting the standard compliance activities required by this document and by other software-related requirements in the project. This planning must be consistent with system-level planning and shall include all applicable items in the SDTP. In particular it must describe the standard compliance test cases and test plans.

The standard compliance testing in EMI-1 is included in the appropriate integration testing plans. The following table lists the compliance testing tasks for EMI-1, affected components, and the associated test plan.

Compliance Testing Technical Area Relevant Components of PTs Plan

GLUE Compute A-REX, CREAM, U. TSI, U. XNJS, UAS-C, WSRFlite, WMS, libarcclient, arc*, UCC, HILA

[R17]

GLUE Data DPM, StoRM, dCache, UAS-D [R17]

SRM Data dCache (client and server) [R18]

HTTP(s) Data DPM, dCache, StoRM [R19]

"file://" access protocol Data DPM, dCache, StoRM [R19]

6.2 SOFTWARE DEVELOPMENT ENVIRONMENT This section describe how the integration, interoperability and standard compliance testing methods and tools described in Section 5 are instantiated in practice for the development and testing of the Products in EMI-1.

6.2.1 Software Engineering Environment The test development activities of the identified EMI components are summarized in the table below. It is shown which EMI product is using which software engineering environment. In particular it describes:

a. The ETICS configurations to be used. If not yet known for EMI-1 the appropriate ETICS component is listed.

b. The target platforms (It is not yet decided which target platforms will be used for EMI-1 at this stage, so the currently available platforms are listed.)

c. Specific versions of programming or scripting languages or development tools to be used (if available per platform).

d. Specific version of external software packages to be used as far as known (if available per platform).

Product ETICS Component/ Configuration

Platforms with languages and development tools

Development tools / Programming languages

External Dependencies (links to list)

A-Rex emi.arc.arc1 All platforms supporting Java. Built by Maven. Current target platforms: Deb5 (64bit), SL5 (64bit)

Maven / Java

+

Make / C

Maven controlled dependencies (no ETICS dependencies)

WMS emi.wms (current conf: emi-

SL4 Make / C https://twiki.cern.ch/twiki/bin/view/EMI/JobManPT



wms_B_3_3) ExternalDeps

CREAM emi.cream-ce (current conf: emi-cream-ce_B_1_13)

SL5, Deb5 Maven / Java https://twiki.cern.ch/twiki/bin/view/EMI/JobManPTExternalDeps

UNICORE CE

emi.unicore.unicorex (current conf: emi.unicore.unicorex.unicore-unicorex_B_TRUNK)

All platforms supporting Java. Current target platforms for EMI-1: Deb5 (64bit), SL5 (64bit)

Maven / Java Maven controlled dependencies (no ETICS dependencies)

ARC* Client

Missing, but expected

Missing, but expected Missing, but expected


UCC Client emi.unicore.ucc (emi.unicore.ucc.emi-unicore-ucc_B_TRUNK)


Maven / Ant / Java Maven controlled dependencies (no ETICS dependencies)

DPM emi.lcgdm (many singular ETICS components)

missing (given later) Missing, but expected

https://twiki.cern.ch/twiki/bin/view/EGEE/DMComponentsList#Dependencies_List

dCache Server

emi.dcache.server (emi-dcache-server_R_HEAD)

SL5 Ant / Java Missing, but expected

dCache Client

emi.dcache.srmclient (mi-dcache-srmclient_R_HEAD)

SL5 Ant / Java Missing, but expected

StoRM emi.storm (emi-storm-backend_R_1_6)

SL5 Ant / Java https://twiki.cern.ch/twiki/bin/view/EMI/StoRMPTExternalDeps

UNICORE data

emi.unicore.unicorex (current conf: emi.unicore.unicorex.unicore-unicorex_B_TRUNK)


Maven / Java Maven controlled dependencies (no ETICS dependencies)

6.2.2 Software Test Environment This section describes the specific environment where testing activities must take place. The environment for each product is listed in the table below. In particular it describes:

a. The ETICS project configurations to be used.

b. The target platforms. For EMI-1 SL5 is the mandatory platform. Probably Debian 5 will also be target platform.

c. The reference testbed configuration to be used and any required endpoint as far as known from the EMI testbed activity [R2]



d. Specific test formats or test tools to be used (possibly per platform)

e. Specific version of external software packages to be used (possibly per platform)

Product ETICS Component/ Configuration

Target Platforms

Testbed availability

Test formats/tools External Software Packages

A-Rex emi.arc.arc1 SL5, (Deb5)

Testbed OS: SLC5.3 x86, Debian Lenny x8

Available

Version 1.1

n/a In development

Planned for Integration tests: Unit tests

Planned for compliance tests: GSTAT [R20]


WMS emi.wms (current conf: emi-wms_B_3_3)

SL5, (Deb5)

Testbed OS: SL4

Available (3 instances)

Version gLite 3.1

n/a In development



https://twiki.cern.ch/twiki/bin/view/EMI/JobManPTExternalDeps

CREAM emi.cream-ce (current conf: emi-cream-ce_B_1_13)

SL5, (Deb5)

Testbed OS: SLC4 x86_64, SL5 x86

Available

Production Version (LSF) 3.2; 3.1.24 Production

n/a In development



https://twiki.cern.ch/twiki/bin/view/EMI/JobManPTExternalDeps

UNICORE CE

emi.unicore.unicorex (emi.unicore.unicorex.unicore-unicorex_B_TRUNK)

SL5, (Deb5)

Testbed OS: openSuSE 11.3

Available

6.3.1 Production

n/a In development




ARC* Client

missing SL5, (Deb5),

Testbed OS: Ubuntu Hardy i386

Available

Release 0.8.2 Production

n/a In development




UCC Client

emi.unicore.ucc (emi.unicore.ucc.emi-unicore-ucc_B_TRUNK)

SL5, (Deb5)


Available

6.3.1 Production

n/a In development




DPM emi.lcgdm (many

SL5, Available n/a In development https://twiki.cern.ch/twiki/bin/view/EGEE



singular ETICS components)

(Deb5)

Testbed OS: SLC4.8 x86

Production 3.2.5


/DMComponentsList#Dependencies_List

dCache Server

emi.dcache.server (emi-dcache-server_R_HEAD)

SL5, (Deb5)

Testbed OS: SL5

Available

Version A + B

n/a In development



dCache Client

emi.dcache.srmclient (mi-dcache-srmclient_R_HEAD)

SL5, (Deb5)

Testbed OS: SL5

Available

Version A + B

n/a In development



StoRM emi.storm (emi-storm-backend_R_1_6_0_5)

SL5, (Deb5)

Testbed OS: SL5

Available

3.1.0-0_ig50_sl4 Production Version

n/a In development


https://twiki.cern.ch/twiki/bin/view/EMI/StoRMPTExternalDeps

UNICORE data

emi.unicore.unicorex (current conf: unicore-unicorex_B_TRUNK)

SL5, (Deb5)


Available

6.3.1 Production

n/a In development



6.2.3 Software Integration Tasks Tracking This section describes the change management tracker instance to be used to track EMI-1 integration tasks.

The integration and testing tasks are tracked at JRA1.7 [R1] and the superior JRA1 Wiki page [R3]. In particular all testing strategies can be found under [R3] since they are related to the general development of the Technical Objectives in JRA1. An overview of the state of the art integration and interoperability statuses is available under the JRA1.7 page. There will be also a table available which summarizes all integration and testing tasks with links to external reports in one piece of information. Columns in this table will indicate and track the current status of the integration testing.

6.3 INTEGRATION, INTEROPERABILTY AND STANDARD COMPLIANCE TASKS This section describes the expected input and output of the testing activity for each Product within the scope of the EMI-1 Integration Plans. The section is organized in subsections, one subsection for each Product. Each subsection contains a description of the integration, interoperability and standard compliance test cases and their implementation. If the information is available in external documents, it is added by reference. If information is not available from the PT, it marked as expected, but missing. If it is not applicable in general or in this development phase, it is marked as ‘n/a’ (Not Applicable).



6.3.1 ARC Computing Element (A-REX) This ARC computing element (CE) implements job execution functions used for a large variety of computational resources.

The following technical objectives are addressed by this component also providing insights in the adopted test plans and the expected outcome of the testing activities. Constraints are reported where given.

Technical Objectives

Test Plans Expected Outcome Constraints RC Date

Compute - Glue 2.0 support in job management services and client tools

Integration: Testing the integration of GLUE 2.0 in A-REX

A-REX component is able to expose GLUE2 compliant pieces of information about computational resources.

n/a 2011-02-28

Interoperability/Standard Compliance: Testing compliance with other GLUE 2.0 enabled compute components

All affected EMI compute components are compliant with GLUE 2.0 and will be interoperable in respect to this schema.

6.3.1.1 Integration testing The testing strategy for the GLUE 2.0 support for EMI compute elements is outlined at [R17]. The integration testing strategy is still under development, and may be refined until EMI-1.

The expected availability of the test implementations is: 2011-01-31

The expected deployment within the EMI testbed is: 2011-01-31

6.3.1.2 Standard compliance testing The EMI compute elements which are supporting GLUE 2.0 should be interoperable in regard to this information schema. The standard compliant testing for GLUE 2.0 in regard to compute elements is still in evaluation. It is planned to create a test suite with using the GSTAT tool.

6.3.2 WMS The WMS receives requests for a job execution from a client, finds the required appropriate resources, then dispatches and follows them until completion.





Integration: Testing the integration of GLUE 2.0 in WMS

WMS component is able to expose GLUE2 compliant pieces of information about computational resources.

n/a 2011-02-28

Interoperability/Standard Compliance: Testing

All affected EMI compute components are compliant



compliance with other GLUE 2.0 enabled compute components

with GLUE 2.0 and will be interoperable in respect to this schema.



The expected deployment within the EMI testbed is: 2011-01-31.


6.3.3 CREAM This service is a simple, lightweight service for job management operations and represents the CE of gLite. The following technical objectives are addressed by this component also providing insights in the adopted test plans and the expected outcome of the testing activities. Constraints are reported where given.




Integration: Testing the integration of GLUE 2.0 in CREAM

CREAM component is able to expose GLUE2 compliant pieces of information about computational resources.

Dynamic GLUE2 information might be experimental (info provider problem)

2011-02-28



6.3.3.1 Integration testing The testing strategy for the GLUE 2.0 support for EMI compute elements is outlined at [R17]. The integration testing strategy is still under development, and may be refined until EMI 1.






6.3.4 UNICORE compute elements The UNICORE compute services provide several Web services that can be used for computational job submission and management including data-staging. It consists of the following components: U.TSI, U.XNJS, WSFRLite, and another internal component that is the common information provider (CIP). The following technical objectives are addressed by this component also providing insights in the adopted test plans and the expected outcome of the testing activities. Constraints are reported where given.




Integration: Testing the integration of GLUE 2.0 in UNICORE compute elements.

UNICORE component is able to expose GLUE2 compliant pieces of information about computational resources.

n/a 2011-02-28

Interoperability/Standard Compliance: Testing compliance with other GLUE 2.0 enabled compute components.






6.3.5 ARC* Client This component is a client solution capable of working with the ARC computing Element and several storage elements. The libarclient is one library integrated within the component.







Integration: Testing the integration of GLUE 2.0 in ARC* client compute elements.

ARC* client components are able to expose GLUE2 compliant pieces of information about computational resources.

n/a 2011-02-28







6.3.6 UCC Client The UCC is the command-line client of the UNICORE technology.





Integration: Testing the integration of GLUE 2.0 in UCC client.

The UCC component is able to expose GLUE2 compliant pieces of information about computational resources.

n/a 2011-02-28









6.3.7 DPM

The Disk Pool Manager (DPM) is a lightweight solution for disk storage management (ease of installation, configuration, low effort of maintenance), while providing all required functionality for a grid storage solution (support for multiple disk server nodes, different space types, multiple file replicas in disk pools, etc.)




Data 1 - All storage elements publishing initial GLUE 2.0 storage information.

Integration: Testing the integration of GLUE 2.0 in DPM.

DPM component is able to expose GLUE2 compliant pieces of information about storage resources.

GLUE2 support is experimental while GLUE1.3 remains production

2011-02-28

Interoperability/Standard Compliance: Testing compliance with other GLUE 2.0 enabled data components

All affected EMI data components are compliant with GLUE 2.0 and will be interoperable in respect to this schema.

Data 3 - All storage elements offering support for the http(s) protocol.

Integration: Testing the integration of the HTTPS support in DPM.

DPM component should be able to use HTTP(S)

n/a 2011-02-28

Interoperability/Standard Compliance: Testing compliance with other HTTPS enabled data components

All affected EMI data components are compliant with HTTPS and will be interoperable in respect to this protocol.

Data 4 - All storage elements offering support for the "file://" access protocol.

Integration: Testing the integration of the file protocol in DPM.

DPM component will be able to support the file:// protocol via NFS4.1

Experimental 2011-02-28

Interoperability/Standard Compliance: Testing compliance with other file enabled data components

All affected EMI data components are compliant with the file protocol and will be interoperable in respect to this protocol.



6.3.7.1 Integration testing The testing strategy for the GLUE 2.0 support of EMI data elements is outlined at [R17]. The integration testing strategy is still under development, and may be refined until EMI-1.

The testing strategy for the HTTPS support of EMI data elements is outlined at [R18]. The integration testing strategy is still under development, and may be refined until EMI-1.

The testing strategy for the file protocol support of EMI data elements is outlined at [R19]. The integration testing strategy is still under development, and may be refined until EMI-1.



6.3.7.2 Standard compliance testing The EMI data elements which are supporting GLUE 2.0, HTTPS, and the file protocol should be interoperable in regard to these standards. The standard compliant testing for these schemas and protocols in regard to data elements is still in evaluation.

6.3.8 dCache

System for storing and retrieving huge amounts of data, distributed among a large number of heterogeneous server nodes, under a single virtual file system tree with a variety of standard access methods.





Integration: Testing the integration of GLUE 2.0 in dCache.

dCache component is able to expose GLUE2 compliant pieces of information about storage resources.

n/a 2011-02-28



Data 2 - Using https instead of httpg for the SRM protocol as a prototype implementation in one storage element and client (library).

Integration: Testing the integration of https in dCache.

dCache server and client can work together without the use of httpg and the GSI.

n/a 2011-02-28

Data 3 - All storage elements

Integration: Testing the integration of the

dCache component should n/a 2011-02-



offering support for the http(s) protocol.

HTTPS support in dCache.

be able to use HTTP(S) 28

Interoperability/Standard Compliance: Testing compliance with other HTTP(S) enabled data components

All affected EMI data components are compliant with HTTP(S) and will be interoperable in respect to this protocol.


Integration: Testing the integration of the file protocol in dCache.

dCache component will be able to support the file:// protocol via NFS4.1

Experimental

2011-02-28




The testing strategy for replacing httpg in SRM protocol is included in [R18]. dCache already performs tests with a SRM compliance test suite on a regular basis. Here it makes sense to broaden the usage of this test suite to other SRM-compliant systems such as DPM and StoRM. The integration testing strategy is still under development, and may be refined until EMI-1.

The testing strategy for the HTTP(S) support of EMI data elements is outlined at [R18]. The integration testing strategy is still under development, and may be refined until EMI-1.




6.3.8.2 Standard compliance testing The EMI data elements which are supporting GLUE 2.0, HTTP(S) in SRM protocol, HTTP(S) in general and the file protocol should be interoperable in regard to these standards. The standard compliant testing for these schemas and protocols in regard to data elements is still in evaluation.

6.3.9 StoRM

StoRM is a service compliant with the open standard Storage Resource Manager (SRM) v2.2 being able to manage any POSIX file systems supporting the Access Control List (ACL) mechanism, including high performance storage systems based on cluster file system (such as GPFS from IBM and Lustre from Oracle/Sun Microsystems).







Integration: Testing the integration of GLUE 2.0 in StoRM.

StoRM component is able to expose GLUE2 compliant pieces of information about storage resources.


2011-02-28



Data 3 - All storage elements offering support for the http(s) protocol.

Integration: Testing the integration of the HTTPS support in StoRM.

StoRM component should be able to use HTTP(S)

n/a 2011-02-28

Interoperability/Standard Compliance: Testing compliance with other HTTPS enabled data components

All affected EMI data components are compliant with HTTPS and will be interoperable in respect to this protocol.


Integration: Testing the integration of the file protocol in StoRM.

StoRM component will be able to support the file:// protocol via GPFS and Lustre.

Experimental 2011-02-28




The testing strategy for the HTTPS support of EMI data elements is outlined at [R18]. The integration testing strategy is still under development, and may be refined until EMI-1.




6.3.9.2 Standard compliance testing The EMI data elements which are supporting GLUE 2.0, HTTPS, and the file protocol should be interoperable in regard to these standards. The standard compliant testing for these schemas and protocols in regard to data elements is still in evaluation. For the GLUE 2.0 compliance it is planned to create a test suite with using the GSTAT tool.



6.3.10 UNICORE data The UNICORE – Data component stands for a flexible framework that is able to integrate various different storage implementations and concepts.





Integration: Testing the integration of GLUE 2.0 in StoRM.

The UNICORE data component is able to expose GLUE2 compliant pieces of information about storage resources.


2011-02-28



Data 5 - File Catalogue Access from UNICORE

Integration: Testing the integration of the File Catalogue Access.

A new UNICORE-Data component element will be developed that can access file catalogues

n/a 2011-02-28


The testing of the File Catalogue Access will likely be performed just by unit tests which will be outlined in the components Validation and Certification Plan. The development of a unit test suite is still in development like the functionality itself.



6.3.10.2 Standard compliance testing No standard compliance testing scheduled for EMI-1.

6.4 OTHER SOFTWARE DEVELOPMENT ACTIVITIES

6.4.1 Risk Management This section describes any identified risks and the proposed corrective actions. Any risks associated with the software development in regard to Technical Objectives are described in the SDTP [R2].

Concerning integration tasks and testing the main risk is that required test suites are not in place at the deadline for EMI-1 testing phase that is the 28th of February 2011. Therefore, JRA1 will continuously gathering and monitoring the testing activities in collaboration with the appropriate Product Teams. Especially, the Validation and Certification Plan of the affected components should be available



before the deadline exceeds. In case the test suites and plans are after all not available JRA1.7 will escalate the issue to JRA1 task leader who will address it to the technical area leaders.

Another risk would be that the test suites and plans are available but the respective hard- and software resources at the EMI testbed are not yet in place. JRA1.7 will avoid that issue by identifying the needed resources for the integration tests until the end of February and checking their availability in the testbed. In general, installing and configuring the required resources shouldn’t be a big obstacle, since the Product Teams have usually co-workers in their teams who are also responsible for the respective parts if the EMI testbed.

A minor risk is a bad flow of information between Product Teams and JRA1. Also JRA1.7 depends on input about the current status of development, schedules, or arbitrary drawbacks in the software engineering process. JRA1.7 tries to avoid this issue by requesting information from the product teams in an early stage of EMI 1. Moreover, ARC has designated a dedicated person who is coordinating all tasks in EMI concerning integration and testing.



7 SCHEDULES AND ACTIVITY NETWORK This section provides a GANTT-based overview of the components which has been taken from the JRA1 EMI-1 Development and Test Plan [R10]. This chart shows software developments relevant for the EMI 1 release.



Figure 3 describes the sub-tasks of JRA1.7 and its relationships to other sub-tasks as well as other WP activities within the EMI project in an activity network. All these sub-tasks essentially collaborate with the JRA1.8 quality control task that in turn provides pieces of information about the test availability and relevant software quality control metrics. The collaboration with task JRA1.8 ensures that JRA1.7 is in-line with quality requirements monitored by the SA2 activity. The direct interaction of JRA1.7 with SA2 is ensured via the extensive use of the EMI testbed.

Figure 3: Activity network



8 CONCLUSIONS This document describes the first year work plan for the integration tasks of the EMI project. This plan is based on the best information at the time and the general plan set out in the parallel overall Technical Development Plan document. The first year integration work plan activities in brief: • During the first months of the project, the work on the integration tasks was focused on the understanding of the latest state-of-the-art of EMI software components in terms of integration statuses and plans. Excel matrices were produced to maintain and monitor current integration and interoperability statuses of EMI components. • In respect of the EMI 1 release, relevant integration tasks were identified according to the EMI Technical Development Plan. • The document describes and defines the principles of different integration and interoperability strategies covered by this plan in terms of system integration and testing, unit integration and testing, and interoperability testing. • A detailed work plan was produced describing the test plans of each EMI component with particular focus on the EMI 1 release. It can be stated that some test plans were still under evaluation when writing this document. For the the EMI 2 integration work plan a more restrictive schedule must be agreed.

DJRA1.6.1 – Integration Work Plan and Status Report

Documents

Transcript of DJRA1.6.1 – Integration Work Plan and Status Report