CS Communication & Systèmes Applied Grid For SMEs [email protected].

CS Communication & Systèmes

Applied Grid For SMEs

[email protected]

Workshop on Grid Utilization - IAS – 20/10/2005 - 2

• Long-term partner for our customers in design, integration and operations of critical applications, systems and infrastructures

• Focus on applications at the heart of our customers' business : expertise, commitment, innovation, sense of service and ability to manage large and complex projects

+3,300 employees (70% engineers)

Headquarters and Regional offices in France6 subsidiaries in EuropeSubsidiaries in Canada, USA, Brazil and Chile

2004 Consolidated Sales : € 348 Millions


Intelligent Transport Systems

Cross-Disciplinary Offers

Banking and Finance

Energy, Utilities and Telecom

Public Administrations

Defence and Military Aviation

Aeronautics and Automotive

Space and Civil Aviation

Critical Systems Integration

Outsourcing &Transitioning IS

NetworkServices

Industrial & CriticalApplications

Critical Real Time software

Mission and Control Centres

Simulation and Scientific Computing

Technical Information Systems

(PLM, EDMS, EAI, Web-services, Virtual

Reality)

Security & Business Continuity

Subsidiaries•SCOT

•Germany•Canada•Romania

Air Traffic Control Centre

Simulator

CommunicationSystems

IntelligenceC4I

Logistic InformationSystems

Toll Systems &Contact less toll

collection

Traffic Management

Subsidiaries•United States

• Chile• Croatia

DistributedSystems

Networks

End-to-End Production

DataMigration

Systems Conversion

Application Maintenance

Architecture and Design

Integration and Deployment

Maintenance and Supervision

Security

Digital Video Surveillance

Application Management / Security / Linux & Open-source Software / Grid Computing

Infrastructure ServicesSystems and Applications

Business skills adapted to our clients needs

Applied Grids for SMEs


Computing and storage facilities

Security and confidentiality

Capacity to use a set of dedicated applications closed to their industrial needs

Costs reduction by mutualisation and computing on demand

Externalization of IT tasks

‘User friendly’ interfaces to HPC infrastructures

Business model matching to their financial capacities

Industrial Applications driven Grids

RUGBI : To SMEs of Biotechnology Industry

Openplast : To SMEs of Polymer Processing Industry

SMEs Needs and Challenges


Application driven Grid Challenges• Needs

• Non computing specialist users.

• Light Client with grid frontier (portal).

• Information System for user request, management and exploitation.

• Technologies Web compliant.

• Quality Of Services (QoS)• Transparency.

• Security / Confidentiality.

• Performance / Permanence.

• Easy business oriented increase and scalability.

• Interoperability, Evolution compliant, Portability.

• Administration / Exploitation / Maintenance.

• Distributed Architecture.

• Open source and Open file format solutions.

• Initial state of the art• DataGrid, E-Toile, GRASP, …


Two Grid projects gathering:

OpenplastRUGBI

Funded by the French Ministry of Research

IT Services company

Software Editor

Business oriented interface

Academic institution

Infrastructure provider


Academics & IT services company :

Computing cluster MPI

Academics & IT services company : Storage bay

IT services company:

Controllerserver

IT services company: Exploitation

server

Actors driven Architecture

Client

SMEs :End - User

Technological park & Editors & Academics :

Web PortalBusiness oriented


High speednetwork

ECP [Paris]PEP [Oyonnax]

CS SI [Grenoble]

CC-IN2P3 [Lyon]

Client

INSA [Oyonnax]LPC

[Clermont Ferrand]

Grids Sites Infrastructures

Openplast RUGBI

Controller server

Web Portal serverComputing Cluster

Storage BayExploitation Server

CC-IN2P3 [Lyon]

CS SI [Grenoble]

High speednetworks

ECP [Paris]

Client

IBCP [Lyon]

Client

Academicinstitutions

BioPôle [Clermont limagne]


The grid portal must provide a user friendly interface for grid management, exploitation and easy use.

User management

Resources management

Sites management

Workflow managementFiles management

Exploitation services

Grid management

Jobs management

Grid

Portal


Resources management services :

• Applications (commercial or not): • Creation.

• Deployment.

• Configuration.

• Update.

• Databases (commercial or not): • Creation.

• Deployment.

• Configuration.

• Update.

• Storage resources with specific roles:• Transit (all uploaded data) on portal.

• Repository (all permanent data) on storage element.

• Working (for job submission) on computing element.


CS GRID Controller : multi-layer architecture

Operating System

Batch Manager

Globus

Grid Services

ControllerInformation System Data

DBMSXML-DB XML Files

Soft & DBdeployment

User Resources Rights Execution/Workflow

Location/Allocation

Filesmanagement

Exploitation

Web Portal

Bu

sin

ess o

rien

ted

ap

plicati

on

s

Bu

sin

ess o

rien

ted

ap

plicati

on

s

Generic layers Specific layers

Controllers provide services to manage users, resources and users rights on resources.

Controllers provide allocation services to allocate jobs execution.

Controllers provide exploitation services : accounting, monitoring, …


• Globus 2.4, 3.2, and now 4.0

• Java (servlet, CoG)

• Web Services SOAP

• OpenSource schedulers (PBS, Condor, …)

• Exchange format : XML

• Information System hosted on DBMS Oracle / Mysql / Xindice wrappers

Technological choices


Computing

Storage

Controller

Exploitation

Portal

Client

• Client Web• Client SOAP• Client CoG Globus GridFtp, MyProxy• Client Mail

•Web Server• Client WS-SOAP•GridFtp Server

• WS-SOAP Server• CoG Globus Gram, GridFtp, MyProxy Clients • SGDB-XML Server

• WS-SOAP Server • MyProxy Server • NTP Server • CA Server • Mail Client

• GridFtp Server • GRAM Server • SOAP Server

• GridFtp Server

FIREWALLSOAP [8080]MyProxy [7521]

FIREWALLGRAM-SOAP [8080]GridFtp [2811]

FIREWALLGRAM-SOAP [8080]GridFtp [2811]

DMZ

FIREWALLSOAP [8080]

FIREWALLSOAP [8080]

FIREWALLGridFtp [2811]HTTPS [8443]

Grid internal Communication

Client external Communication

Portal-Controller Communication

Grid Network Architecture

Workshop on Grid Utilization - IAS – 20/10/2005 - 15 Internet provider Storage and Computing Network

Intranet CS Network

DMZ

CS Node Network Architecture compliant with internal and external grids



Storagebay


StoragebayController

server

Exploitationserver

Site A

Site B

User portal

Connection

Authorization

Authentication

proxy creation

certificate creation

Grid basic job submission principles

Web Portalserver



Storagebay

Site B

User portal

File



server

Exploitationserver

Site A

File

Job submission

Workflowsettings

Fileupload

Web Portalserver

Workflowgeneration

Workflowexecution

File transfer

File

Jobexecution




Storagebay

Site B

User portal



server

Exploitationserver

Site A

Web Portalserver

Workflowexecution

File

Jobexecution

result

Files transfer to user spaceWorflow engine informs user

Updates portal/

send mail


Screen Shots of generic grid functions


Examples : Installation packages Controller configuration

Intuitive Grid Portal/Controller installation & configuration


Examples : Grid-client Files Management User management

User friendly interfaces


Exploiting and Monitoring• Subscription forms

• Statistical use diagrams

Specific tools


RUGBI Tools and databases

Automatically Updated Databases

Protein knowledgebaseSwissprotTrembl

3D protein structure dataPDB

Encyclopedia of genes and genomesKEGG

nucleotide sequence database EMBL

compounds diversity set NCI

Tools

Sequence alignmentBlastFasta

Multiple sequence alignmentClustalw

Protein 2D structure prediction PredatorSympa96GorIV…

Docking of compounds to proteinsAutodock

Metabolic Pathway Analysis


RUGBI alignment tool: the BLAST example


Platform for polymer injection molding simulation• Project management : case-studies oriented

• Allows to specify all the stages of polymers injection modeling

• Qualifying and Validating each stage of the process

CADCAD

ProjectProject

MeshMesh

SubprojectSubproject

Datasettings

Datasettings

StudyStudy

MoldingMolding

CaseCase

AnalysisAnalysis DefectsDefectsStudy view

Project view

• Data enrichment

Mesh Soft.

Mesh Soft.

SolverSolver Post-treatment

Post-treatment

CAD +Mesh

Plast3DPlast2DPlastTherm3DPlastPart2D

PamViewPamRTM

Geomesh

CAD +Mesh +DS [LC, IC, FC]

PlastDefects

XML

Data settings

Data settings AnalysisAnalysis

PlastPre

CAD CAD +Mesh +DS +Raw results

CAD +Mesh +MED +Raw res. +Analysed res.

CAD +Mesh +DS + Raw res. +Analysed res. +Post-treated res.

CATIA

DSDS : : Data Data settingssettings

XMLXML XMLXMLXML XML XML


Study Framework

Prototype Openplast client modules

Post-treatment framework


• Applied grids for SMEs :• Security based on Globus authentication by proxy and certificate.

• Confidentiality managed by a strict use of user directories (temporary, working, storage,…) based on user rights.

• Exploitation and Accounting facilities for different user type.

• Easy of use for non grid specialists.

• Basic set of applications (client and grid side).

• Expected Grid End-Users• RUGBI

• IBCP portal gets daily 2000 connections that will be transferred on RUGBI nodes.• Different SMEs from Clermont Limagne BioPole and others.

• Openplast• Polymer processing industry from which 70% are SMEs.• European Polymer Institute.

Conclusion


• Starting point: the SMEs real needs

• Business Interface partners

• Openplast : European Polymer Institute

• RUGBI : BioPôle Clermont Limagne.

• In progress: • RUGBI: study market with Lyon Biopôle, Arteb, Théogone …• Openplast: partnership with PEP.

• IT service company (IN2P3, CS).

• Academic institutions for their up-to-date research knowledge.

• Software editors: Openplast: ESI, Fluent, … & RUGBI: Accelrys, Tripos, …

• Infrastructure: extend from a regional level to national

• RUGBI: Theogone and different biopoles and interoperability with future national french bioinformatics grid.

• Openplast: HPC pole - Iles de France (IOLS).

• European: EGEE2 (functional test OK with GT2.4, and interoperability with Glite in progress).

For a good industrial accomplishment

Contacts

http://rugbi.in2p3.frhttp://www.openplast.org

Project manager : [email protected]

Questions and Answers ?

More Technical slides


USER_IDUSER_TYPE

USER_IDLOGINPWD

CERTIFICATION_SUBJECTCERTIFICATION_CREATION_DATE

CERTIFICATION_END_DATECERTIFICATION_RENEW_DATE

CERTIFICATION_AUTHORITYCERTIFICATION_TYPE

STREETCITY

ZIPCODELAST_NAMEFIRST_NAME

CREATION_DATEEND_DATE

STATE

USER_IDDESCRIPTION

NAMECREATION_DATE

END_DATESTATE

1..*

0..*

RIGHTS

USER_IDRESOURCE_ID

RWXSTATE

PERIODPRIORITY

QUOTA

RESOURCES

RESOURCE_IDSITE_ID

RESOURCE_TYPE

0..*

0..*

RESOURCE_IDSIZES

DESCRIPTIONFORMAT

DATA_TYPE

RESOURCE_IDNAME

DESCRIPTIONVERSIONEDITOR

ADMINISTRATOR_NAMEXML_FILE_NAME

RESOURCE_IDNAMESIZESURL

DESCRIPTIONVERSIONFORMAT

ADMINISTRATOR_NAMEXML_FILE_NAME

DATAS SOFTWAREDB

SITE

NAMECITY

SITE_IDCOMPANY

DESCRIPTIONXML_FILE_NAME

ADMINISTRATOR_ID

SITE

NAMECITY

SITE_IDCOMPANY

DESCRIPTIONXML_FILE_NAME

ADMINISTRATOR_ID

0..*

10..*

USERS

1..*

1

0..*

GROUPS CERTIFIED

Manage users, groups and rights on resources.

Manage logical resources : softwares and databases.

Manage physical resources : Computing / Storage elements, Portals …

Information System 1/3


Database updating RUGBI service allows :• Releases updates

• Versions updates

• Versions management

• Versions deployment

FTPserver

Available files list

XML FILE

Comparison

Metadata(names, date)

Available files list

XML FILE

Missing files

Download

- Waiting end of jobs on old database- Deletion of the old database

- Replication- Registration

Update



StoragebayMaster

Controllerserver

Exploitationserver

Site B

Web Portalserver


Storagebay

SlaveController

server

Site A

Web Portalserver

datamodification


Storagebay

SlaveController

server

Site C

Web Portalserver

User portal

dataModificationnotification

dataModificationnotification

dataupdates

Grid information system update mechanisms



StoragebayMaster

Controllerserver

Exploitationserver

Site B

Web Portalserver


Storagebay

SlaveController

server

Site A

Web Portalserver


Storagebay

SlaveController

server

Site C

Web Portalserver

User portal

dataupdate

notification

dataupdates

dataupdates

Grid information system update mechanisms


Controller

Interface<<DataRequestor >>

MySQL

XIndice

Wrapper

MySQLRequestor

Wrapper

XIndiceRequestor

DataRequestorFactory

Dump.xml

Requests Backup

Restore

Wrappers :

MySQL, Oracle, PostGres, XIndice compliant.

Backup / Restore throw XML files driven by the master controller.

Information System 3/3


GRID

Workflow

EngineApplication developer

Submit jobs, transferfiles, invoke grid services, receive notification events, etc.

While:- Monitoring execution- Handling errors- Recovering workflow failures- Freeing allocated resources- etc.

End-user

Chooseapplications, input files, arguments,etc.

WebPortal

s+

Controller

job1

job2

jobs

executionenvironment

GRID : End-user interface / Application developer interface


<graph wf:NAME="graph-8433" xmlns:wf="http://www.in2p3.fr/workflow/language" xmlns:x0="http://www.w3.org/2001/XMLSchema"> <wf:job wf:NAME="job-8433" wf:contact="sirius.si.c-s.fr:2119/jobmanager-fork" wf:isBatchJob="false" wf:provider="GT2">

<wf:executable wf:shell="/bin/sh" wf:stageIn="true" wf:tmp="/grid/transit/user/1/projects/8433/"> <![CDATA[/grid/software/BLAST/2.4/bin/blastall

-p blastn -i $INPUT -d /grid/db/ecoli/ecoli.aa -o $OUTPUT -v 500 > $OUTPUT]]></wf:executable><wf:sandbox wf:intermediary="gsiftp://sirius.si.c-s.fr:2811/grid/vo/CS-VO/cs-user1/projects"> <wf:input wf:VAR="INPUT" wf:source="file:///grid/transit/user/1/projects/8433/input.in"

wf:destinationdir="$HOME/tmp/8433" > <wf:output wf:VAR="OUTPUT" wf:destination="file:///grid/transit/user/1/projects/8433/output.out"

wf:sourcedir="$HOME/tmp/8433"/></wf:sandbox>

</wf:job> <wf:mail wf:NAME="mail-8433" wf:contact="mail.grid.fr" wf:depends="job-8433"

wf:from="[email protected]" wf:subject="your job 8433 is finished." wf:to="[email protected]"> <wf:message> <![CDATA[ Your results are on your transit space on CS grid portal CS ... ]]></wf:message> </wf:mail></graph>

Workflow used spacesPortal transient space (input and output) /grid/transit/user/1/projects/8433Temporary CE space gsiftp://sirius.si.c-s.fr:2811/grid/vo/CS-VO/cs-user1/projects/Temporary Working node space $HOME/tmp/8433CE software location /grid/software/BLAST/2.4/bin/blastall :CE database location /grid/db/ecoli/ecoli.aa

Execution sequence executed by the workflow engine

Execution sequence example

CS Communication & Systèmes Applied Grid For SMEs [email protected].

Documents

Transcript of CS Communication & Systèmes Applied Grid For SMEs [email protected].