11:00 – 11:20UNICORE – A European Grid Middleware (20 min) Achim Streit (FZJ)

A European Grid MiddlewareJanuary 2007Achim [email protected]

*Forschungszentrum Jlich

History lessonUNiform Interface to COmputing Resourcesseamless, secure, and intuitiveInitial development started in two German projects funded by the German ministry of education and research (BMBF)08/1997 12/1999: UNICORE projectResults: well defined security architecture with X.509 certificates, intuitive GUI, central job supervisor based on Codine from Genias 1/2000 12/2002: UNICORE Plus projectResults: implementation enhancements (e.g. replacement of Codine by custom NJS), extended job control (workflows), application specific interfaces (plugins)Continuous development since 2002 in several European projectsOpen Source under BSD license since Summer 2004Core developers today from Europe: CINECA, ICM, Intel, FLE, FZJ


Key featuresA vertically integrated Grid middleware system since 1997Provides seamless, secure, and intuitive access to distributed resources and dataUsed in production and projects worldwideOpen Source under BSD licenseFeaturesintuitive GUI with single sign-onX.509 certificates for AA and job/data signingworkflow engine for complex workflowsextensible application support with plug-insinteractive access with UNICORE-SSH


ProjectsMore than a decade of German and European research & development and infrastructure projectsUNICORE


Recent DevelopmentsInteractive access (UNICORE-SSH)Improved workflow capabilities (MetaPlugin for Workflows)High-level API for programming Grids (Roctopus)DRMAA-based TSI Collaborative Online Visualization and Steering (COVS)Comfortable configuration toolSite Functionality Monitoring Tool (SIMON)


Website: http://www.unicore.eu


Usage at the National German HPC center NICAbout 450 users in 200 research projects of them uses UNICORE Access via UNICORE toIBM p690 eSeries Cluster (1312 CPUs, 8.9 TFlops)

SoftComp Cluster (264 CPUs, 1 TFlop)JUGGLE (176 cores, 845 GFLops)Cray XD1 (120 CPUs + FPGAs, 528 GFlops)


Distributed European Infrastructure for Supercomputing ApplicationsConsortium of leading national HPC centers in EuropeDeploy and operate a persistent, production quality, distributed, heterogeneous HPC environmentIDRIS CNRS, FranceFZJ, Jlich, GermanyRZG, Garching, GermanyCINECA, Bologna, ItalyEPCC, Edinburgh, UKCSC, Helsinki, FinlandSARA, Amsterdam, NLHLRS, Stuttgart, GermanyBSC, Barcelona, SpainLRZ, Munich, GermanyECMWF, Reading, UK


ServicesDedicated 1 Gb/s network as a basisHigh performance datagrid via GPFSExtended to non-AIX Linux like SGI Altix, Mare Nostrum Common Production Environment on all sitesJob migration across sitesUsed to load balance the global workflow when a huge partition is allocated to a DEISA project in one siteUNICORE as Grid Middleware for workflow applicationsCo-allocation for applications running on multiple sites at the same timeGlobal data management to include tertiary storage and hierarchical data management systemScience Gateways and Portals to facilitate the access of new, non traditional users communities


Usage in DEISAfully-meshed UNICORE infrastructurecomplex multi-site workflows easily possibleheavily used by scientific DECI (DEISA Extreme Computing Initiative) projects/jobs


Usage in D-GridCore D-Grid sites committing parts of their existing resources to D-GridApprox. 700 CPUsApprox. 1 PByte of storageUNICORE is installed and usedAdditional Sites receiving extra money from the BMBF for buying compute clusters and data storageApprox. 2000 CPUsApprox. 2 PByte of storageUNICORE (as well as Globus and gLite) will be installed as soon as systems are in place


Usage in Industry and Commercial Support


Roadmap to UNICORE 6.0New infrastructure based on web servicesPreserved traditional User-level featuresAtomic: simple tasks, such as Execute scriptClient: workstation GUI Workflow: edit, run and monitor graphs of atomic tasksAdditional User-level featuresPortal: web based portal clientStreaming: client-server streaming support (for visualization or media applications)Application development featuresSoftware license managementSimplified application deploymentDeployment featuresUser and virtual organization (VO) management


Architecture of Version 6.0


Interoperability and Usability of Grid InfrastructuresGoal is to provide key software components for building e-infrastructuresProject will demonstrate proof of concept with expectation for a follow-on project in FP7 Initial focus on providing common interfaces and integration of major Grid software infrastructuresCommitted to standards processImplementing agreed open standards and working with standards process (OGF/Oasis)


inCommon interoperable servicesDatabase Access (OGSA-DAI), VO Management (VOMS), Accounting (GGF-RUS), Job Submission and Monitoring (OGSA-BES, ESI/UAS), Portal (Gridsphere)Capability to add additional servicesEmphasis on porting and re-engineering work, not developing from scratchInfrastructure integrationInteroperable security frameworkInitial gLite/Globus/UNICORE/CROWNgrid interoperabilityQuality AssurancePublished methodology and compliance testRepository of open-source, quality assured software services for gLite, Globus, UNICORE and CROWNgridEvaluation infrastructure to test services


inProvide an integrated Grid solution for workflow-centric, complex applications with a focus on data, semantics and knowledgeBased on UNICORE 6End user focus: ease of use, domain specific tools, hidden GridFocus on data services and knowledge management solutionsProvide decision support services for risk assessment, toxicity prediction, and drug design9 Partners (from PL, DE, EE, NI, IT, FR, CH), coordinated by ICMFilling an existing gap for science and businessPortalsIntegration of a established commercial portal solution, EnginFrame and an open source solutionA-WARE Workflow Orchestrator Service (WOS)A flexible and powerful workflow engineMaturing and support of UNICORE 6 technology5 partners (from IT, UK, FR, DE), coordinated by CINECA


A use case of UNICORE 6Collaborative Online Visualization and Steering (COVS)A framework for scientific simulations and visualizationsIn addition to usual post-processing (offline) techniquesEnables to view the actual status (online) of parallel simulationsBased on the communication library VISITWorks with all VISIT-enabled scientific visualizationsImplemented as a higher level service of UNICORECollaboration server + multiplexer for geographically dispersed clientsUsage of UNICORE security infrastructure for single sign-onSuccessfully demonstrated at OGF18, Europar06, SC06,

WS-RF macht Session Management, SSH fr DatentransferCollaboration Server fr viele ClientsKein Transfer von Bilder, auf den Clients wird das Bild gerechnet !Collaboration Server + VMUX wird durch UNICORE konfiguriertWS-RF Franework in Anwendung: Ressourcen ist Collaboration Server und VMUX, der durch einen UNICORE-Services gesteuert wird

11:00 – 11:20UNICORE – A European Grid Middleware (20 min) Achim Streit (FZJ)

Documents

Transcript of 11:00 – 11:20UNICORE – A European Grid Middleware (20 min) Achim Streit (FZJ)