EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

EMI 2 Matterhorn roadmap

Balázs Kónya (Lund University)EMI Technical Director

EGI Technical Forum, Lyon, 20 September 2011

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

outline

EMI is about software, the mission of the project is to deliver software for DCIs

•EMI development cycle•Development highlights•EMI 2 ”Matterhorn”

– Content– Timeline

Without dwelling into the details

20/09/2011 EGI TF, Lyon 2

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

EMI on the DCI landscape

20/09/2011 EGI TF, Lyon 3

EMI

DNA1.4 - EMI Roadmap and DCI Collaborations

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

First peak has been conquered: EMI 1

20/09/2011 EGI TF, Lyon

Kebnekaise (2104 m), 12 May 2011

4

Support to be provided until 28.2.2013

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Next peak to climb: EMI 2

20/09/2011 EGI TF, Lyon

Matterhorn (4448 m), April 2012

5

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611We have started the journey

20/09/2011 EGI TF, Lyon

• We defined the Plan (what to climb, how to go, what to carry) • DNA1.3.2 ”Technical development plan”

• We have the necessary tools• build & integration, task tracking

• We are on the way

6

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

From requirements to released products

20/09/2011 EGI TF, Lyon 7

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611While climbing to the peak ....

• In addition to EMI 2 development, EMI has a large ”base camp” to take care of

• Maintenance– bug fixing

• Support – issue resolution

• The ”base camp” has the highest priority

20/09/2011 EGI TF, Lyon 8

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Matterhorn: requirements

• Sources of requirements– EGI-TCB: formal requests communicated via EGI Tracker – WLCG: informal requests communicated verbally or via email– Users: direct communication with PTs

• Handling of requirements– Requirement collection and processing for EMI 2: CLOSED– No major changes in EMI 2 directions

• Last minute case-by-case adjustments are possible

– Requirement collection for EMI 3: OPEN• Requirement assessment is a continously running process

• Everything is recorded and tracked:– https://savannah.cern.ch/task/?group=emi-req – note: requirements from the project proposal phase are not there

20/09/2011 EGI TF, Lyon 9

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Matterhorn: requirementsEndorsed top priority requirements:•WMS should provide better feedback about jobs, automated resubmission of stuck jobs•Release EMI clients on SL5/32•Standard file locations for logs and temporary files•Publish mw service version•Provide Nagios probes for EMI services•Better error messages (more verbose, informative)•Coherency of command line commands parameters•Coherency of APIs•Support for Debian 6•Support for SL6•Ubuntu support at least for EMI clients•CEs should support a set of LRMS•Reduce memory consumption of CREAM

20/09/2011 EGI TF, Lyon 10

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Matterhorn: requirements

and some non-accepted ones:•Fixing known bugs before adding new features•Uniform logging•Improving (automatization) yaim configuration•Unicore GUI improvements

20/09/2011 EGI TF, Lyon 11

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

EMI Workplan (3 years)

20/09/2011 EGI TF, Lyon 12

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611Dev highlights: compute (1/2)

• Implementation of the agreed common job management methods (EMI-ES interface)– CREAM, UNICORE, A-REX Computing Elements– All EMI compute clients

• Complete GLUE2 support– GLUE2 used in resource discovery, brokering, matchmking

• Agreement and support for an EMI compute accounting record (UR)• ARGUS integration of computing elements• Lots of product-level improvements

– see next slide as an example

20/09/2011 EGI TF, Lyon 13

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611Dev highlights: compute (2/2)

gLite compute PT plans:•CREAM High Availability

– Pool of CREAM CEs seen as a single logical CE

•(S)GE support for CREAM•WMS Support for DAG jobs without using Condor DAGMan

– This will enable the submission to CREAM CEs also for DAG jobs

20/09/2011 EGI TF, Lyon 14

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Dev highlights: data (1/2)

• GLUE2 support in data clients• EMI Data Access Library design and implementation• Storage Element and catalogue synchronization

– Will not be released with EMI 2

• Better UNICORE data integration– SRM, LFC, AMGA

• NFS 4.1 / pNFS for EMI storage elements• focus on standard protocols and procedures

– POSIX file access and integration into SRM (File://)– http/WebDAV/Cloud storage

details: Monday Data Workshop presentations20/09/2011 EGI TF, Lyon 15

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

20/09/2011 EGI TF, Lyon 16

Dev highlights: data (2/2)

What we actually want

ROOTROOT

GoogleStorage

AmazonS3

CloudFront

EMIStorage

FTS 3Gs,S3REST,

WebDAVHTTP

Gs,S3REST,

WebDAVHTTP

EMIStorage

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Dev highlights: security

• Finalization of ”EMI profiles”– Common SAML profile– Common XACML profile

• Security Token Service (STS) service– Simplified management of security credentials (AAI)– Authenticates and authorizes users based on security tokens– Transforms the security token into another security token– Implementation based on Shibboleth IDP & OpenWS/SAML v3 (Shib3)

• EMI Authentication Library implementation– JAVA, C, C++ APIs

• EMI delegation agreement– A delegation solution to replace ”GSI” in compute and data areas

Details: Wednesday ”EMI Security” session

20/09/2011 EGI TF, Lyon 17

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Dev highlight: infra

• Full utilization of GLUE2 including validation

• EMI Service Registry• Delivery of service

monitoring via NAGIOS– Each EMI service should

come with probes

• Modified accounting publishers– DGAS, APEL– Accomodate new records– Utilize messaging

20/09/2011 EGI TF, Lyon 18

EMI Service Registry

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Dev highlight: common/cross area

• Cloud strategy (see next talk)• Consolidation plans for each area

– What can be phased out?

• Additional platforms– Debian 6 (with an eye on latest Ubuntu)– SL6

• Usability improvements – Command line parameters– Error messages

20/09/2011 EGI TF, Lyon 19

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Matterhorn: what will be inside the box?• EMI 2 will be largely based on products taken from EMI 1

– EMI 1 contains 54 products from ”AMGA to WMS”

• Product updates with possible backward incompatible changes

• New products:– STS, Pseudonimity – EMI Service Registry, EMI_datalib, EMI_authlib

• Components to be phased out– Not yet decided, consolidation plans are due in October– There’ll be products tagged as ”discontinued”– EMI project is contractually obliged to cut its portfolio

• The content will be finalized as part of the release preparation– We are strict on deadlines, delayed products can and will drop out

20/09/2011 EGI TF, Lyon 20

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Matterhorn: release timeline (1/2)

• Development tasks defined and being tracked - July, 2011• Some task are not yet recorded

• BUILD environment ready for all platforms: – August, 2011• Not fully completed

• QA policies and procedures ready – September, 2011

• RC1 (1st code freeze) – 14th Dec, 2011– Successful build and packages available– All foreseen functionalities should be implemented

• RC2 (2nd code freeze) – 1st Feb, 2012 – Bug fixes– Available test reports for in-house deployment of RC1

20/09/2011 EGI TF, Lyon 21

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Matterhorn: release timeline (2/2)

• RC3 ( 3rd code freeze) - 29th Feb, 2012– RC2 functionality and config testing, deployment on EMI testbed

• RC4 (final release) - 28th Mar, 2012– Full test reports for RC3– Fully deployed RC3 on the EMI testbed– Only authorized bug fixes (tracked)

• Final Release available in testing repository - 30th Mar, 2012– Complete testing and certification reports available– All tasks “Certified”

• Announce EMI 2 (Matterhron) - 26th Apr, 2012

• Preparation, planning for EMI 3 starts – January 2012

20/09/2011 EGI TF, Lyon 22

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Matterhorn in a nutshell

• The next Major release from the EMI project– Contains appx. 50-60 integrated products for

compute, data, security and infra capabilities– Backward incompatibility allowed!

• The most development-dense EMI release• Will be available on multiple platforms• Planned release date: 26 April 2012• Support will run well after the end of EMI

project

20/09/2011 EGI TF, Lyon 23

EMI I

NFS

O-R

I-261

611

EMI I

NFS

O-R

I-261

611

Thank you

20/09/2011 EGI TF, Lyon

EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611

24