EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

EMI 1 and WLCG

Francesco Giacomini (INFN) - SA1Morris Riedel (JSC) – JRA1

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• EMI 1 Release Plan– Francesco Giacomini (INFN)

• EMI 1 Developments– Morris Riedel (JSC)

2011-02-09 WLCG GDB, CERN 2

Outline

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

2011-02-09 WLCG GDB, CERN 3

Release Plan

Start EMI 1 EMI 2 EMI 3

Support & Maintenance

Support & Maintenance

Support & Maintenance

Supp. & Maint.Major releases

01/05/2010 30/04/2011 30/04/2012 28/02/2013

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

WLCG GDB, CERN 4

• Periodic major releases of the distribution– Tentatively once per year– Right place to introduce new features and

backwards-incompatible changes– Minimize number of changes within a major release

• Clear separation between maintaining what is in production and introducing new features– SA1 vs JRA1– People (developers, testers, etc.) are the same, but

the rules are different in the two contexts

2011-02-09

Stability vs Innovation

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

2011-02-09 WLCG GDB, CERN 5

Release Plan

Start EMI 1 EMI 2 EMI 3

Support & Maintenance

Support & Maintenance

Support & Maintenance

Supp. & Maint.Major releases

01/05/2010 30/04/2011 30/04/2012 28/02/2013

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Two EMI major releases supported at any time→Lifetime of an EMI major release is two years– Full support: 18 months– Security only: another 6 months– Applies to previous mw distributions as well• e.g. gLite 3.2 and arc 0.8.x would be fully

supported until October 2011, with security updates until April 2012

2011-02-09 WLCG GDB, CERN 6

Support

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Guiding principles: simplification and adoption of established best practices

• EMI 1 reference platform: SL5/64 + EPEL– SL6 and Debian available in EMI 2, maybe sooner already during EMI 1 for

some components• One EMI repository (per major release) for all services

– base + updates + security• Following EPEL packaging policies

– e.g. installation in /usr, not in /opt/glite– Full compliance may be achieved later

• Metapackages just for installation– Cannot be used to version the installation of a certain product

• In general, no upgrade path between major releases– Including from gLite 3.2 to EMI 1

2011-02-09 WLCG GDB, CERN 7

Deployment Highlights

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

2011-02-09 WLCG GDB, CERN 8

EMI 1 Developments At a Glance

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Development impacts: Integrated Release– EPEL-compliant component installation (/usr)– Change of Globus from VDT to EPEL

(IGE project officially supports the EPEL version)• Key Developments– VOMS using standard SSL (instead of GSI)– Started migration towards GLUE2 • Improved information validation checkings

– Improvements for Data components• NFS4.1, WebDAV, better performance, etc.

2011-02-09 WLCG GDB, CERN 9

EMI 1 Developments At a Glance

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

2011-02-09 WLCG GDB, CERN 10

EMI 1 Developments – Security

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• VOMS– Drop of the dependencies on Globus Toolkit• Will simplify the installation (+ smaller size)

– Completed integration of VOMRS functionalities in VOMS-Admin• (VOMRS itself is going to end support)

– Usability improvements: Better error messages, cleaner handling of invalid inputs

– REST interface to obtain VOMS ACs• No major features in security components 2011-02-09 WLCG GDB, CERN 11

EMI 1 Developments – gLite Security

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

2011-02-09 WLCG GDB, CERN 12

EMI 1 Developments – Compute

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• CREAM-CE (CREAM, BLAH, CEMON)– Experimental GLUE 2 support • Working on solutions with information providers• GLUE1.3 still supported

– ARGUS Integration for both CREAM and CEMon• Config: Using ARGUS or existing authZ with gJAF

– Better support for multi core allocations • (i.e. JDL attributes WholeNodes, HostNumber, etc.)

2011-02-09 WLCG GDB, CERN 13

EMI 1 Developments – CREAM

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Workload Management System (WMS)– Work in progress: Adoption of GLUE2, but in

WMS needs to be further investigated– Manage GLUE1.3 and GLUE2 in JDL unclear– Support for 'feedback‘• Jobs 'stuck' in a queue for too long can be

migrated on another Computing Element• At configuration time: feature must be enabled to

work as described

2011-02-09 WLCG GDB, CERN 14

EMI 1 Developments – WMS

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• EMI 1 A-REX based ARC CE– End of life of

Grid Manager– The same interfaces are kept• In addition: New Web service interface

– Existing features kept– New Features: plugins, improved input/output

data handling, GLUE2 information publishing, new security framework

– ARGUS support (experimental)2011-02-09 WLCG GDB, CERN 15

EMI 1 Developments – ARC CE

http://www.nordugrid.org/arc/gm-eol.html

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

2011-02-09 WLCG GDB, CERN 16

EMI 1 Developments – Data

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• dCache – Initial GLUE2 support for exposing storage information

in parallel to GLUE1.3– NFS4.1 Support (client comes free most likely in SL6)– WebDAV Support – Support for ARGUS blacklisting– Significantly improved configuration system• Already tested at selected Tier 1's and Tier 2's.

– More details, tutorials and hands-on will be provided during the "5. dCache Workshop" in Goettingen (16/17 March)

2011-02-09 WLCG GDB, CERN 17

EMI 1 Developments – dCache

http://indico.desy.de/conferenceDisplay.py?confId=3959

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• File Transfer Service (FTS)– Support for resuming gridftp transfers where the

endpoints allow it

• Disk Pool Manager (DPM)– GLUE2 Support exposing storage information

(gfal/lcg_util support expected shortly after EMI 1)– Improved performance in dpm-drain

(parallel file replication)

2011-02-09 WLCG GDB, CERN 18

EMI 1 Developments – FTS & DPM

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

2011-02-09 WLCG GDB, CERN 19

EMI 1 Developments – Infrastructure

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• No major new features in BDII– Information flow largely based on GLUE1.3

moving towards GLUE2• But Information content improvement– Low confidence level that the values for

some attributes are correct– Better information providers are needed in the

overall set of relevant components– Better test tools available that we work on• GLUE2 validation, GStat tests

2011-02-09 WLCG GDB, CERN 20

EMI 1 Developments – Information

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

2011-02-09 WLCG GDB, CERN 21

EMI 2 Outlook

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

• Solution for storage elements and catalogue synchronization

• Common Storage Accounting Records– Input to standardization activities

• Common EMI Execution Service Interface– Preserve backwards-compatibility

• Common Security Profiles– SAML&XACML: Common Attribute Profiles

• Experimental adoption of messaging2011-02-09 WLCG GDB, CERN 22

EMI 2 Outlook

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Thank you

23

EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611

2011-02-09 WLCG GDB, CERN