Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) [email protected]. 2 Outline...
Transcript of Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) [email protected]. 2 Outline...
![Page 2: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/2.jpg)
2
OutlineOutline
Motivation & HistoryProduction UNICORE 5
FeaturesArchitectureExample deployments within DEISA and T-Systems
UNICORE 6Improved FeaturesArchitecture & Higher Level ServicesFuture Developments
UNICORE as Open SourceUNICORE ForumSummary
![Page 3: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/3.jpg)
3
Motivation: Scientists & Parallel ComputingMotivation: Scientists & Parallel Computing
Scientists need computational and storage related resources
InterconnectingNetwork
Data
![Page 4: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/4.jpg)
4
Supercomputers are managed by Resource Management Systems (RMSs) that handle the scheduling
But: There are many RMSs available
All proprietary way of job submitIBM Loadleveler llsubmit…Torque Resource Manager qsub…Different job description languages…(# of nodes, memory requirements,…)
Motivation: Scientists & Parallel ComputingMotivation: Scientists & Parallel Computing
Resource Management System
![Page 5: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/5.jpg)
5
Solution: Grid System UNICORE
Define job workflows in abstract mannerImmediate portability of job definitionsfor other systems with other architecturesNo ‘learn overhead’ if a new RMS is usedApplications across multiple supercomputers/clusters ‘going meta’
Motivation: Scientists & Parallel ComputingMotivation: Scientists & Parallel Computing
![Page 6: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/6.jpg)
6
Initial UNICORE Initial UNICORE developmentsdevelopments
UNICORE 08/1997-12/1999
UNICORE Plus 01/2000-12/2002
EUROGRID 11/2000-01/2004
GRIP 01/2002-02/2004
OpenMolGRID 09/2002-02/2005
![Page 7: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/7.jpg)
7
From Testbed to ProductionFrom Testbed to Production
Success factor: VERTICAL INTEGRATION
2006Different communitiesDifferent computing resources (super computers, clusters, …)Know-how in Grid middleware
2002
![Page 8: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/8.jpg)
8
2008200720062005200420032002 2009
UniGridsDGI
VIOLA
OMII-EuropeEGEE-II
A-WARE
GRIPGRIDSTART
OpenMolGRID
ChemomentumeDEISA
DEISA
UNICORE in (mostly) European ProjectsUNICORE in (mostly) European Projects
![Page 9: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/9.jpg)
9
Production UNICORE FeaturesProduction UNICORE Features
![Page 10: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/10.jpg)
10
OGSA-based and WSRF-compliant UNICORE 6 alphaInteractive access UNICORE-SSHHigh-level API for programming GridsDRMAA-based access to RMSsComfortable configuration tool
Recent DevelopmentsRecent Developments
![Page 11: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/11.jpg)
11
UsiteUsite VsiteVsite
UNICORE Architecture OverviewUNICORE Architecture Overview
TSI
NJS
RMS
TSI
NJSAuthorization
Gateway Authentication
opt. Firewall
Gateway
opt. Firewall
Client
Multi-Site Jobs
UUDB
SSL
Abstract
Non-Abstract
Disc RMS Disc
Vsite
TSI
NJS
RMS
UUDB
Disc
IDBIDB IDBIncarnation
opt. Firewall
Authorization
similar to /etc/grid-security/grid-mapfile
Workflow-EngineResource ManagementJob-MonitoringFile TransferUser ManagementApplication Support
similar to Globus jobmanagerforkLoadLeveler, (Open)PBS(Pro), CCS, LSF, NQE/NQS, ...CONDOR, GT 2.4
![Page 12: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/12.jpg)
12
UNICORE ClientUNICORE Client
JobPreparation
JobMonitoring
WorkflowManagement
Usites
Vsites
![Page 13: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/13.jpg)
13
UNICORE 5 Architecture (1)UNICORE 5 Architecture (1)
Definition of abstract Jobs (not machine-specific)Creation of complex workflows (e.g.multi-site jobs for geographically dispersed supercomputers)
Usite B
Vsite Y
Vsite Z
client tier
UNICORE CLI
UNICORE Client
AJOand
UPL over SSL
Non-abstract Jobs
Resource Management
System
Target System
Interface
target system tier
Gateway authentication
server tier
optional firewall
submitand / or
execution
server tier
IDB
UUDB authorizationand
incarnation
NetworkJob
Supervisor
ScientificApplications
Abstract Jobs
schedulingand
resource management
Usite C
Vsite X
Multi-site Jobs
Usite A
![Page 14: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/14.jpg)
14
UNICORE 5 Architecture (2)UNICORE 5 Architecture (2)
Client extensions via application-specific pluginsCar-Parrinello Molecular Dynamics Plugin(speed up quantum chemical computations)Fluent, Gaussian, MSC Nastran PluginsOpenMolGrid – Plugins (Molecular Science)
UNIX-Style Command-Line Interface (CLI)
Usite B
Vsite Y
Vsite Z
client tier
UNICORE CLI
UNICORE Client
AJOand
UPL over SSL
Non-abstract Jobs
Resource Management
System
Target System
Interface
target system tier
Gateway authentication
server tier
optional firewall
submitand / or
execution
server tier
IDB
UUDB authorizationand
incarnation
NetworkJob
Supervisor
ScientificApplications
Abstract Jobs
schedulingand
resource management
Usite C
Vsite X
Multi-site Jobs
Usite A
![Page 15: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/15.jpg)
15
UNICORE 5 Architecture (3)UNICORE 5 Architecture (3)
Various plugins have been developed the last yearsBase upon the Arcon Client API (Java - API)
Usite B
Vsite Y
Vsite Z
client tier
UNICORE CLI
UNICORE Client
AJOand
UPL over SSL
Non-abstract Jobs
Resource Management
System
Target System
Interface
target system tier
Gateway authentication
server tier
optional firewall
submitand / or
execution
server tier
IDB
UUDB authorizationand
incarnation
NetworkJob
Supervisor
ScientificApplications
Abstract Jobs
schedulingand
resource management
Usite C
Vsite X
Multi-site Jobs
Usite A
![Page 16: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/16.jpg)
16
Virtual Organization
UNICORE 5 Architecture (4)UNICORE 5 Architecture (4)
GatewaySingle entry-point for UNICORE Sites (Usites)Contains n Virtual Sites (Vsites)(e.g. provides access to n supercomputers)Only ONE OPEN PORT in firewallAuthentication of users via X.509 certificatesPerfect base to group resources and accesswithin Virtual Organizations (VOs)
GATEWAY
Usite B
Vsite Y
Vsite Z
client tier
UNICORE CLI
UNICORE Client
AJOand
UPL over SSL
Non-abstract Jobs
Resource Management
System
Target System
Interface
target system tier
Gateway authentication
server tier
optional firewall
submitand / or
execution
server tier
IDB
UUDB authorizationand
incarnation
NetworkJob
Supervisor
ScientificApplications
Abstract Jobs
schedulingand
resource management
Usite C
Vsite X
Multi-site Jobs
Usite A
![Page 17: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/17.jpg)
17
UNICORE 5 Architecture (5)UNICORE 5 Architecture (5)
Network Job Supervisor (NJS)Analyse defined Workflows (workflow engine)
Multi-site, multi-step JobsDirected Acyclic Graphs
Submit of sub-jobs to other sites over Gateway
Job Incarnation via Incarnation Database (IDB)turn abstract-jobs into machine-specific jobsE.g. location of preinstalled software packages
Authorization via Unicore User Database (UUDB)turn X.509 user certs to users xlogin of machineThe complete certificate is checked (not only DN )
Usite B
Vsite Y
Vsite Z
client tier
UNICORE CLI
UNICORE Client
AJOand
UPL over SSL
Non-abstract Jobs
Resource Management
System
Target System
Interface
target system tier
Gateway authentication
server tier
optional firewall
submitand / or
execution
server tier
IDB
UUDB authorizationand
incarnation
NetworkJob
Supervisor
ScientificApplications
Abstract Jobs
schedulingand
resource management
Usite C
Vsite X
Multi-site Jobs
Usite A
![Page 18: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/18.jpg)
18
UNICORE 5 Architecture (6)UNICORE 5 Architecture (6)
Target System InterfaceRepresents a real Target System(e.g. a supercomputer or storage server)
Interaction with Resource Management SystemsTorque, PBS Pro, Load Leveler, (UNIX fork), …Job submit to Resource Management SystemScheduling via Resource Management SystemRetrieve job results and job outputManage workspace
Access to pre-defined Applicationse.g. a massive parallel simulation for collaborative visualization & steering via UNICOREUsite B
Vsite Y
Vsite Z
client tier
UNICORE CLI
UNICORE Client
AJOand
UPL over SSL
Non-abstract Jobs
Resource Management
System
Target System
Interface
target system tier
Gateway authentication
server tier
optional firewall
submitand / or
execution
server tier
IDB
UUDB authorizationand
incarnation
NetworkJob
Supervisor
ScientificApplications
Abstract Jobs
schedulingand
resource management
Usite C
Vsite X
Multi-site Jobs
Usite A
![Page 19: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/19.jpg)
19
TSI & Resource ManagementTSI & Resource Management
Target System Tier Consists of one Target System InterfaceInteractions with underlying RMSs
Target System InterfaceEach supported RMS is used in conjunction with a target system-specific TSI impl. in perlPerl TSI implementations are stateless daemonsStatically configured before startup (RMS admin)Various TSI implementationsPBS TSI (Linux), LSF TSI (SGI), PBS-Pro (HP-UX), NQS (IRIX), LoadLeveler TSI (AIX), …
Usite B
Vsite Y
Vsite Z
client tier
UNICORE CLI
UNICORE Client
AJOand
UPL over SSL
NJS-TSI protocol
Gateway authentication
server tier
optional firewall
server tier
IDB
UUDB authorizationand
incarnation
NetworkJob
Supervisor
ScientificApplications
Abstract Jobs
Usite C
Vsite X
Multi-site Jobs
Usite A
Text-based
submitand / or
execution
schedulingand
resource management
Resource Management System
target system tier
PBS ProTSI (HP-UX)
PBSTSI (Linux)
perlCCSTSI
LoadLevelerTSI (AIX)
...TSI
NQSTSI (IRIX)
NoBatchTSI
Target System InterfacesLSF
TSI (SGI)
Resource ManagementSystem Commands
![Page 20: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/20.jpg)
20
TSI Framework using DRMAATSI Framework using DRMAA
Base upon Open Source SUN Grid Engine DRMAA impl.Java Bindings of DRMAA specification are used
Java-Based Target System Interface CoreOnly one layer of UNICORE is exchanged protocols still the same
Platform & RMS command independentDRMAA - API “only” provides Interfacesfor job submission & management
Additional DRMAA Extensions for file transferTransfer over Unicore Protocol Layer (UPL)Optional transfer over GridFTP
UNICORE 5/6
![Page 21: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/21.jpg)
21
Consortium of leading national HPC centers in EU
IDRIS – CNRS, FranceFZJ, Jülich, GermanyRZG, Garching, GermanyCINECA, Bologna, ItalyEPCC, Edinburgh, UKCSC, Helsinki, FinlandSARA, Amsterdam, NLHLRS, Stuttgart, GermanyBSC, Barcelona, SpainLRZ, Munich, GermanyECMWF, Reading, UK
UNICORE in DEISA (1)UNICORE in DEISA (1)
![Page 22: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/22.jpg)
22
InterconnectingNetwork
Data
Deploy and operate a persistent, production quality, distributed, heterogeneous supercomputing environment
UNICORE in DEISA (2)UNICORE in DEISA (2)
![Page 23: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/23.jpg)
23
Fully-meshed UNICORE infrastructure among partnersComplex multi-site workflows easily possibleHeavily used by DECI projects
UNICORE in DEISA (3)UNICORE in DEISA (3)
![Page 24: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/24.jpg)
24T-Systems Solutions for Research GmbHHPC / Grid ServicesAlfred Geiger
UNICORE in Business scenarios
![Page 25: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/25.jpg)
25
UNICORE UNICORE -- LIFE LIFE -- CDCD
Complete “out-of-the-box” usage of UNICORE 5Bootable Linux OS with UNICORE 5 pre-installedDoes not harm your system Sandbox scenarioFor testing, evaluating, …
AVAILABLE HERE AND AT THE
GRID VILLAGEUNICORE BOOTH
![Page 26: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/26.jpg)
26
Internet
Firewall
UNICORE Gateway
Compute Resources
Network Job Supervisor (NJS)VO=DGI
DRMAA-based Java Target System Interface (TSI) Core
DRMAA-compliant Globus TSI
GRAM Client GridFTP Client
Globus 2GRAM
Gatekeeper
GRAM Job-ManagerResource
Management System
UNICORE
GridFTPServer
Uspace
MDS
UUDBIDB
Initial Interoperability Initial Interoperability UNICORE & UNICORE & GlobusGlobus 2.42.4
UNICORE-Client
StandardizationnecessaryProprietaryProtocols
UNICORE AJO/UPL, Globus RSL
![Page 27: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/27.jpg)
27
Emerging Grid Standards (1)Emerging Grid Standards (1)
OGSA – Open Grid Services Architecture“The Physiology of the Grid”, GGF - GFD30Resources shared via services in GridsLifecycle management for Web Services necessary
OGSI – Open Grid Services InfrastructureGlobus Toolkit 3 provides early implementation(US Projects using Globus are better funded by US – Gov)
WS-RF - Web Services Resource FrameworkRe-factors and evolves OGSI to exploit WS-* technologiesSame functionality as OGSI, but separated Specifications
![Page 28: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/28.jpg)
28
Emerging Grid Standards (2)Emerging Grid Standards (2)
Change of TerminologyFrom Grid Services …
…to stateful Web Services
![Page 29: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/29.jpg)
29
Development of next generation of UNICORE called UNICORE 6 based on OGSA and compliant with WS-RF
Broader vision of interoperability between different Grid middlewareInteroperable infrastructures through standards
UNICORE 6UNICORE 6
![Page 30: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/30.jpg)
30
HTTP Body doGoogleSearch()
Host-To-Network – Layer : PPP
Understanding SOAP & WSUnderstanding SOAP & WS--RF (1)RF (1)
Using WSDL description for XML tags of doGoogleSearch()
Internet – Layer: IPv4
Transport – Layer: TCP
Application – Layer: HTTP
„Data – Layer“: SOAP
HTTP Header TCP IP Link
<soap><soap:header>
http://api.google.com/search/beta2
</soap:header><soap:body>
doGoogleSearch( key = ‚Grid‘ )</soap:body>
</soap>
e.g. HTTP POST REQUEST with SOAP-compliant doGoogleSearch() call
![Page 31: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/31.jpg)
31
Understanding SOAP & WSUnderstanding SOAP & WS--RF (2)RF (2)
Host-To-Network – Layer : PPP
Internet – Layer: IPv4
Transport – Layer: TCP
Application – Layer: HTTP
„Data – Layer“: SOAP
Host-To-Network – Layer : PPP
Internet – Layer: IPv4
Transport – Layer: TCP
Application – Layer: HTTP
„Data – Layer“: SOAP
Client Google Server (http://api.google.com/search/beta2)
doGoogleSearch()GoogleSearchPort
HTTP POST Request: doGoogleSearch (XML)HTTP POST Response: doGoogleSearchResult (XML)
SearchFunctionalityRegistry / URL
WSDL of Service
1
2
3
4
5
![Page 32: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/32.jpg)
32
Understanding SOAP & WSUnderstanding SOAP & WS--RF (3)RF (3)
Host-To-Network – Layer : PPP
Internet – Layer: IPv4
Transport – Layer: TCP
Application – Layer: HTTP
„Data – Layer“: SOAP
Host-To-Network – Layer : PPP
Internet – Layer: IPv4
Transport – Layer: TCP
Application – Layer: HTTP
„Data – Layer“: SOAP
Client UNICORE Server (http://jump.kfa-juelich.de/TargetSystemService)
submit()TargetSystemPort
HTTP POST Request: submit (XML JSDL)HTTP POST Response: submitResult (XML)
Registry / URLWSDL of Service
1
2
3
4
5
JobExecution
![Page 33: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/33.jpg)
33
StandardizationStandardization forfor interoperabilityinteroperability
Web Services Resource Framework (WS-RF) protocolAdd semantics & syntax to operations (GetResourceProperties)
Get a list of properties that the service is offeringA Web Service itself is typically Stateless WS-RF statefulGrids need access to stateful resources
jobs, supercomputers,telescope, collider,…
Autonomic behaviourServices know how theycan interactStandardized operations & properties Host-To-Network – Layer : PPP
Internet – Layer: IPv4
Transport – Layer: TCP
Application – Layer: HTTP
„Data – Layer“: SOAP
„ Message-Layer“: WS-RF
![Page 34: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/34.jpg)
34
WSWS--RF SpecificationsRF Specifications
Base for others WS-Notifications, WS-Agreement, …Five Specifications ( public comment - Version 1.2 )
WS - Resource Relationship Web service and resource
WS – Resource LifetimeLifecycle of a WS-Resource, TerminationTime, etc.
WS – Resource PropertiesGetters/Setters/Queries of Properties
WS - BaseFaultsBase set of information that appear in fault messages
WS – Service GroupsCollections of WS or WS-Resorces
![Page 35: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/35.jpg)
35
A WSA WS--RF message via SOAPRF message via SOAP(use of WS(use of WS--Addressing&SecurityAddressing&Security))
![Page 36: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/36.jpg)
36
StatefulStateful Web Services & Grids with UNICOREWeb Services & Grids with UNICORE
‚Grid Services‘ – Web Services that implement Grid patternsOpen Grid Services Architecture (OGSA)
WS-RF is one implementation of OGSA conceptsAccess and management of Grid resources of interestWeb Services with standardized state interactions
State of a supercomputer (memory, applications, etc.)State of submitted jobs on resources
Internet
Web-Server
Web-ServerWeb-Client
Web-ClientWeb-Server
Web-Clients
Grid
Supercomp.
Daten
OGSA Grid Dienste
Geräte
Grid Ressourcen
Grid-Clients
Size raise ademand forautonomicbehaviour
OGSA Grid Services
Grid Ressources
![Page 37: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/37.jpg)
37
Workflow Engine
FileTransfer
UserManagement
(AAA)
Brokering ResourceManagement
VISITService
Developments ofUNICORE 6 alpha
Network Job Network Job SupervisorSupervisor
TransformingTransforming UNICORE UNICORE
WS-RF WS-RFWS-RF
WS-RF WS-RFWS-RF
SOAP GatewayWS-RF
Client Portal CommandLine
WS-RF WS-RFWS-RF
Gateway
Host-To-Network – Layer : PPP,…Internet – Layer: IPv4, …
Transport – Layer: TCP, …
JSDL, etc.
Secure Socket Layer: SSLApplication – Layer: HTTPS, …
„Data – Layer“: SOAP,…„ Message-Layer“: WS-RF
![Page 38: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/38.jpg)
38
Job Submission Description Language ExampleJob Submission Description Language Example
<?xml version="1.0" encoding="UTF-8"?> <JobDefinition xmlns="http://schemas.ggf.org/jsdl/2005/11/jsdl"> <JobDescription> <Application>
<POSIXApplication xmlns="http://schemas.ggf.org/jsdl/2005/11/jsdl-posix"> <Executable>/bin/echo</Executable><Argument>hello world</Argument>
</POSIXApplication> </Application>
</JobDescription> </JobDefinition>
JSDL Specification GFD.56 November 2005 - Published as GGF Recommendation
![Page 39: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/39.jpg)
39
Architecture of UNICORE 6.0Architecture of UNICORE 6.0Client
RMS Disc
Gateway
UNICORE 6.0
HigherLevel
Service
GridBeanService
Target System Service
Job Mgmt.
Service
StorageMgmt.
Service
File Transfer Service
Target System Factory
Atomic Services
Globus, OGSA-based Middleware
HigherLevel
Service
GridBeanService
Target System Service
Job Mgmt.
Service
StorageMgmt.
Service
File Transfer Service
Target System Factory
Atomic Services
RMS Disc
RegistrySecurityService
ServicesAPIS GridBean
GridBean API
![Page 40: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/40.jpg)
40
New Gateway of UNICORE 6New Gateway of UNICORE 6
Multiple Protocol SupportSupports UPL
For Production UNICORE Installations
Supports WS-* technologiesWS-AddressingSOAP messages / HTTPS
Retains Unicore Security (!)Single point of entryClient AuthenticationSSL Connections
![Page 41: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/41.jpg)
41
UniGridsUniGrids AtomicAtomic ServicesServices
Interoperability between OGSA based Grid systemsCross–authentication for UNICORE and Globus
Enable cross–Grid resource brokeringWorkflows over different Grid Systems
Unicore Globus OGSAServer
GridClient
GridPortal
TargetSystemService
JobManagement
Service
StorageManagement
Service
FileTransferService
Target System Factory
UniGrids Atomic Services
![Page 42: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/42.jpg)
42
InteroperabilityInteroperability & Standards& Standards
Definition in January 2005Evaluation and refinement in June 2005Execution Service Interfaces document drafted together by UNICORE and Globus developers
Feed in standardisation process of GGF viaOGSA Basic Execution Services (BES) Working Group
TargetSystemService
JobManagement
Service
StorageManagement
Service
FileTransferService
Target System Factory
UniGrids Atomic Services
![Page 43: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/43.jpg)
43
Job Submission StandardsJob Submission Standards
OGSA - Basic Execution Services (BES) Unigrids (Uniform Interface to Grid Services)
Developed WS-RF- based UniGrids atomic services (UAS)UAS: basic interfaces for job/file management in UNICORE 6 alpha
Execution Services InterfaceDraft by Argonne (Globus) and Fujitsu (UNICORE)Consists of UAS input and Globus GRAM demandsInput to the OGSA - BES GGF working group
Refactoring of UAS when OGSA-BES is revised
TargetSystemService
JobManagement
Service
StorageManagement
Service
FileTransferService
Target System Factory
![Page 44: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/44.jpg)
44
Grid Programming Environment (GPE) ClientsGrid Programming Environment (GPE) Clients
GPE consists of a set of Grid toolse.g. GPE Client Framework for UNICORE 6
Client-side for UNICORE 6 and other Grid systemsThree different clients for three different usagesGridBeans as scientific-area specific Plugins
Expert Client Application Client Portal Client
![Page 45: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/45.jpg)
45
GPE Clients and GPE Clients and GridBeanGridBean Deployments Deployments
![Page 46: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/46.jpg)
46
Expert ClientApplication
Client Portal Client
GB
GB GB
GBGB
GBGB
Portable Clients and GridBeans Portable Clients and GridBeans
GridBean
GPE Clients work on top of Unicore/GS, GTK4 or ChinaGridOnly requirement: Atomic Service interfaces
GridBeans are portable without modificationsClient API hides underlying infrastructure details
GB
![Page 47: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/47.jpg)
47
Database Access: OGSA Database Access: OGSA -- DAIDAI
OGSA – Database Access and Integration ServicesInitially implementedwithin UniGrids
Re-engineering ofOGSA-DAI alphaimplementationwithin EU Project OMII - Europe
OGSA – DAIservices hostedin UNICORE 6
Data ServicesData Services
OGSA-DAI Core
RelationalDatabase
XMLDatabase File
Data ServiceResource A
Data ServiceResource B
Data ServiceResource C
WSI Data Service WSRF DataService
Client ToolkitWSI Client Stubs WSRF Client
Stubs
Client Application
SOAPOverHTTP
Client layer
Presentation layer
Business logic layer
Data layer
GridBean
UNIGRIDS ToolkitWSRF Client
Stubs
SOAPOverHTTP
WSRF DataService
![Page 48: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/48.jpg)
48
Database Access: OGSADatabase Access: OGSA--DAI DAI GridBeanGridBean
![Page 49: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/49.jpg)
49
Other Grid Beans for ApplicationsOther Grid Beans for Applications
BLAST GridBean used by molecular biology scientists to determine the structure and sequence of DNAKTA GridBeans provide solutions for an industrial application
Application named as the PreStack Depth Migration (PSDM)Application belongs to the Kirkhoff True Amplitude (KTA) methods class for seismic signal processingRuns on machines with MPI
![Page 50: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/50.jpg)
50
GridFTPGridFTP File Transfer Service ArchitectureFile Transfer Service Architecture
Client – Server transfers
Server – Servertransfers
Including SRB
GFTS GridBeanavailable for clients
Data Transfer of huge files via GridFTP
![Page 51: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/51.jpg)
51
Collaborative Visualization & Steering (1)Collaborative Visualization & Steering (1)
VISualization Interface Toolkit (VISIT)Light-weight library that supports bi-directional data exchange between visualizations and parallel applicationsVisualisation application acts as a server:All operations have to be initiated by the simulationSSH Tunnel for using the VISIT protocol for secure communications with short latency (UGSF optionally)
VISIT / GS FamilyA higher level service family for collaborative visualizationsHosted within the Unicore/GS WS-RF hosting environment Parallel to the atomic services
![Page 52: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/52.jpg)
52
Collaborative Visualization & Steering (2)Collaborative Visualization & Steering (2)
VISIT GridBeanManage participants of visualization sessions via VISIT / GS family
Framework works with all VISIT -enabled applications
Unique Unicore/GS feature for interactive scenarios Not naturally provided by other Grid middleware, e.g. Globus, gLiteUsed in Astrophysics (nBody), Laser-Plasma Physics (PEPC),…
Life Demoat Grid Village
![Page 53: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/53.jpg)
53
Initial Portal Clients & Services (1)Initial Portal Clients & Services (1)
![Page 54: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/54.jpg)
54
Initial Portal Clients & Services (2)Initial Portal Clients & Services (2)
![Page 55: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/55.jpg)
55
Initial Portal Clients & Services (3)Initial Portal Clients & Services (3)
![Page 56: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/56.jpg)
56
Roadmap to UNICORE 6 Production QualityRoadmap to UNICORE 6 Production Quality
New infrastructure based on Web servicesOGSA-based and WS-RF-compliantHeavily committed to Open Standards from GGF & OASISVarious useful and modern higher level services (e.g.streaming)
Alpha was released at the end of July 2006http://unicore.sourceforge.net
TBD: Name des Bundles!
Beta in July 2007
Final in End 2007 (production quality)
![Page 57: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/57.jpg)
57
Improving the knowledge oriented scientific/industrial workflow capabilities & usability
Improving the maturity of atomic and higher level servicesJob Submission OGSA - BESData Access OGSA – DAIAccounting RUS and URVO Management VOMSPortals GridSphere
Improving Interoperability & Interoperation with other Grid systems
Some Future DevelopmentsSome Future Developments
![Page 58: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/58.jpg)
58
http://unicore.sourceforge.net
Open Source under BSD license
Ready-to-use for research and industry
Strong security
Easy installation &configuration
![Page 59: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/59.jpg)
59
http://unicore.sourceforge.net
Academic support by FZJIntegration of own results andfrom other projectsRelease managementProblem tracking, assistanceCVS, Subversion, mailing lists, docs
Viable basis for many other projects
Upcomingnew Website
![Page 60: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/60.jpg)
60
Founded by developers, leading EU HPC centres, and supporting hardware vendors as a non-profit associationFoster the distribution and use of UNICORE, organizeworkshops, support presentations at conferences, publish and maintain the specifications, coordinate further development, certify implementations and extensions
![Page 61: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/61.jpg)
61
SummarySummary
UNICORE is a Grid System from Europe, is open-source and is used in production worldwide (e.g. DEISA)UNICORE 6 base upon OGSA conceptsEmerging standards such as WS-RF implement OGSAUNICORE 6 is compliant with the WS-RF OASIS StandardUniGrids Atomic Service as basic set for job managementUniGrids Atomic Service contribute to ESI & OGSA-BESUNICORE 6 massively commits to emerging Grid standards
OASIS WS-RF&WSN, GGF JSDL, W3C WS-AddressingUNICORE is Open Source … under BSD licenseUNICORE Forum … supports activities after projects
… to meet the UNICORE community
![Page 62: Morris Riedel (FZJ) - unicore.eu · Morris Riedel (FZJ) m.riedel@fz-juelich.de. 2 Outline Motivation & History Production UNICORE 5 Features Architecture Example deployments within](https://reader030.fdocuments.in/reader030/viewer/2022040408/5eba8c6ab25801380f63ca63/html5/thumbnails/62.jpg)
62
http://summit.unicore.org/2006/index.htmlhttp://summit.unicore.org/2005/index.html
VISIT THEGRID VILLAGE
UNICORE BOOTH