Post on 04-Jan-2016
Innovative services and Innovative services and tools for NorduGrid (NGIn)tools for NorduGrid (NGIn)
Nordunet 2008, 11.04.08Nordunet 2008, 11.04.08
Jon Kerr Nilsene-mail: j.k.nilsen@fys.uio.noUniversity of Oslo, Department of PhysicsPhD student, NGIn
04/20/23 www.nordugrid.org 2
The NorduGrid The NorduGrid CollaborationCollaboration
Nordic Grid Neighbourhood
A research A research collaboration that:collaboration that:
Develops and Develops and supports ARC supports ARC middlewaremiddlewareProvides Provides middleware to middleware to research groups research groups and national Grids and national Grids Coordinates the Coordinates the various inputs to various inputs to the KnowARC the KnowARC codecode
A research A research collaboration that:collaboration that:
Develops and Develops and supports ARC supports ARC middlewaremiddlewareProvides Provides middleware to middleware to research groups research groups and national Grids and national Grids Coordinates the Coordinates the various inputs to various inputs to the KnowARC the KnowARC codecode
20/04/23 www.nordugrid.org 3
ARC middleware deployed on 50+ sites in 15+ countries all around the world
http://www.nordugrid.org/monitor
20/04/23 www.nordugrid.org 4
Nordic Data Grid Nordic Data Grid Facility Facility
NDGF pilot project started 2003Goal: Create a Nordic Grid infrastructureBuilds on history and competence of Nordic Grid collaborationNorduGrid/ARC middleware chosen as basisFunded (2 M.EUR/year) by National Research Councils of the Nordic countriesCoordinates and hosts major Grid projects (e.g. the Nordic LHC Tier-1)Operates Nordic storage facility for major projectsDevelops grid middleware and services
NOS-NDK SF N S
Nordic Data Grid Facility
NDGF Facility - 2007Q3NDGF Facility - 2007Q3
20/04/23 www.nordugrid.org 6
Univ. of Oslo NorwayUniv. of Lund SwedenUniv. of Copenhagen DenmarkUniv. of Uppsala SwedenNIIF HungaryGeneva Univ. Hospitals SwitzerlandUniv. of Kosice SlovakiaUniv. of Lübeck Germanyscience+computing ag GermanySUN Microsystems Hungary
NorduGrid members and research teams in medicine, bioinformatics, physics, engineering, automotive industry apps, IT
EU KnowARC project:EU KnowARC project: 10 partners from 7 10 partners from 7 countriescountries
20/04/23 www.nordugrid.org 7
Goals of KnowARC Goals of KnowARC The mission of KnowARC is to create a novel, powerful Next
Generation Grid middleware– extend and re-design ARC– ARC in major Linux distributions– New platforms: Windows, Solaris,
Mac OS-X to promote Grid standardization and
interoperability– interoperability with other Grid
solutions (EGEE and OGF compliant grids)
– possibilities to include (or to be included in) other infrastructures
to prove usage in Health care, Industry and Science– Bioinformatics: Autoimmune
Diseases– Medical Informatics: Lung Diseases– Automotive industry
DENMARK:
University of Copenhagen (NBI, DIKU)
Southern Danish University, Dep. of Computer Science
Aalborg University, Dep. of Computer Science
SWEDEN:
Lund University, Department of Physics
Linköping University, National Supercomputer Center
Umeå University, Department of Computing Science and HPC2N
Uppsala University, Department of Radiation Sciences
Royal Institute of Technology, NADA/PDC, KTH
Stockholm University and KTH, Department of Physics
NORWAY:
University of Oslo, Physics institute
University of Bergen, Department of Physics
University of Tromsø, Department of Computer Science
NTNU, Trondheim, Department of Computer and Information Science
NTNU, Trondheim, IT division (ITEA)
RUSSIA:
St. Petersburg State University, Dep. of Physics
Petrodvorets Telecommunication Center
Dubna?
ESTONIA:
National Institute of Chemical Physics and Biophysics (NICPB)
Estonian Educational and Research Network (EENet)
Tartu University, Institute of Technology
FINLAND:
University of Helsinki, Helsinki Institute of Physics
LITHUANIA:
Vilnius University, Institute of Materials Science and Applied Research
LATVIA
ICELAND
Nordic Grid Neighbourhood
... 9 countries, 24 institutions
Project: … , NO27, Nabo 05-06, Nordic Grid Neighbourhood Project no.: 172724/V11 We have read and approved your final academic and financial project report for the funding period 2005-2006. We are pleased that most of the planned goals of the network have been reached through, i.a. workshops, conferences and tutorials. We are impressed by the number of partners and participants in the various activities.
20/04/23 www.nordugrid.org 9
http://www.nordugrid.org/ngin http://www.nordunet3.org/
•Innovative services and tools for NorduGrid. PI: Farid Ould-Saada, University of Oslo.
4 year grantOslo, Lund, Uppsala, Copenhagen, Helsinki, ReykjavikMainly:
Data Management: storage, indexing and movement Education and expert training in Grid computing
5 PhD grants starting January 2007Grid school
call for collaboration with Baltic and Russia … ??
NGIn for Research and Education
Nordunet3-funded Internet Research Program
NGIn managementNGIn management
20/04/23 www.nordugrid.org 10
Project leaderProf. Farid Ould-Saada, Univ. of Oslo, Norway
Technical coordinatorProf. Alex Read, Univ. of Oslo, Norway
Steering GroupProf. Farid Ould-Saada, Univ. of Oslo, NorwayProf. Paula Eerola, Univ. of Lund, SwedenProf. Tord Ekelöf, Univ. of UppsalaDr. Miika Tuisku, Helsinki Institute of Physics, FinlandDr. Anders Wäänänen, Niels Bohr Institute, Denmark
Participating institutionsPhysics Department, Univ. of Oslo, NorwayPhysics Department, Univ. of Lund, SwedenPhysics Department, Univ. of Uppsala, SwedenDepartment of Radiation Sciences, Univ. of Uppsala, SwedenNiels Bohr Institute, Univ. of Copenhagen and Univ. of Aalborg, DenmarkHelsinki Institute of Physics (HIP), Univ. of Helsinki, FinlandCSC, Finnish IT Centre for Science, Helsinki, FinlandUniv. of Iceland, Reykjavik, Iceland
NGIn - training new Grid NGIn - training new Grid expertsexperts
20/04/23 www.nordugrid.org 11
From the original application: ”The goal of this project is twofold: To extend the excisting middleware and to train new Grid experts...”NGIn obtained 30% of funding applied forThe primary goal of the NGIn project is now to train new Grid expertsTraining programme includes:
Grid PhD studentsA Nordic Grid school (first held in Copenhagen in September 2007)
The NGIn activities will go in line with the other NorduGrid related projectsBudget 2008:
PhD students: 160 k€
Researcer training: 15 k€
Other expenses: 3 k€
Total: 178 k€
NGIn PhD studentsNGIn PhD students
20/04/23 www.nordugrid.org 12
Four PhD students hired, fifth student soon to be hiredIn chronological order:
PhD 1, Jon Kerr Nilsen, Univ. of Oslo, NorwayPhD 2, Mika Silander, Helsinki Institute of Physics, FinlandPhD 3, Salman Toor, Univ. of Uppsala, SwedenPhD 4, Phillippe Gros, Univ. of Lund, SwedenPhD 5, to be announced soon, Univ. of Oslo, Norway
The students are funded 50% by Nordunet3, 50% by local sourcesAll the PhD projects involves data management
Jon Kerr Nilsen, OsloJon Kerr Nilsen, Oslo
20/04/23 www.nordugrid.org 13
Thesis title: Data Management in Grid Middleware
Address limitations and implement solutions in data management and data handling in ARC middlewareUse the solutions in ATLAS data analysisWork in collaboration with both NGIn and the Scientific Computing Group at Univ. of Oslo
Current statusGot familiar with ARC and data management (and Grid acronyms)Survey on data management in conjunction with Grid technology presented at NorduGrid2007Set up a test cluster with dCache and ARC Storage Element (in collaboration with Salman Toor)
Near futureStorage in conjunction with virtualization
Virtualization provides possibilities to run Linux applications in WindowsChallenge to get data out of virtual machine and store them securely and efficiently
Client side of the next generation ARC storage solution
Will work with KnowARC in implementing client-side tools for accessing data in the next generation ARC
Look at data management of local versus external dCache instances in current ARC
Is there any gain in prefering dCache instances in the same network as the computing resource?
Jon Kerr Nilsen, OsloJon Kerr Nilsen, Oslo
20/04/23 www.nordugrid.org 14
27/03/08 www.nordugrid.org 15
Problem: delegation is too coarse-grained in current middleware, e.g. Users delegate all rights (Impersonation) Limited proxy certificates (exclude shell access) Delegating the least set of rights is desirable
Work so far on Analysing ARC components for types of operations
and objects for rights modeling Analysing SAML for extending it to support fine-
grained rights expressions Evaluation of existing solutions with applicable
features, e.g. SecPAL
Thesis title: Fine-grained rights delegation in Grids
Mika Silander, HelsinkiMika Silander, Helsinki
27/03/08 www.nordugrid.org 16
Potential solution candidates SAML + XACML
Pros: other Grid mw committed to these standards
Cons: complexity in policy definition and evaluation, fine-grained rights not modeled
Security Policy Assertion Language (SecPAL) Pros: broad set of security functionality
including rights restrictions, policies expressed with easily understandable syntax
Cons: restrictive license and implementation languages
Mika Silander, HelsinkiMika Silander, Helsinki
27/03/08 www.nordugrid.org 17
Near future plansNear future plans
Near future plans Explore possibility of combining pros of the two
former for ARC: Easily understandable syntax similar to
SecPAL, but, Implementation relying on C/C++ and
adhering to pure SAML & XACML Ensure this is compatible with the security
architecture of ARC v1.x being developed Evaluate SAML + XACML solution of gLite
Applicable, parts reusable?
Salman Toor, UppsalaSalman Toor, Uppsala
20/04/23 www.nordugrid.org 18
Thesis title: Data indexing and data movement in grid environments
Topic: Management of scientific data in Grid systemsMaking a Grid Storage Resource Manager
A layer on top of various Storage Managers (dCache, DPM, Castor, GPFS...)A complete cost based estimation system to find the best SM amongst the SMs registered
Salman Toor, UppsalaSalman Toor, Uppsala
20/04/23 www.nordugrid.org 19
ProgressDeployed dCache test site in Uppsala (with a pool-node in Oslo)Started study of SRM specifications in conjunction with different Storage ManagersDeployed the full Globus toolkit and implemented test services and clients
Next step write a service that
gather the information (owner, permissions etc) of data stored in dCachereplicate the data in a different Storage Manager
Philippe Gros, LundPhilippe Gros, Lund
Physics oriented project become an expert user of AliEn
more Grid oriented project work on the AliEn-ARC integration
Current status: AliEn expertise
still a basic user, which is already a first step(submitting jobs on the Grid)
AliEn-ARC integration project formulated recently studying the current state, existing simple interface
Thesis title: Analysis and management of high energy physics data with NorduGrid
NGIn Grid SchoolNGIn Grid School
20/04/23 www.nordugrid.org 21
The NGIn training programme includes:Grid PhD studentsA Nordic Grid school
The NorduGrid 2007 Grid school was held at Niels Bohr Institute in Copenhagen 26-27 September 2007Organized by CSC, Finland
More than 35 present for the introductory lectures on first dayTutorials and demos had 15 participantsTutorials in topics from High Energy Physics to bio-informatics and medical imagingFrom beginner to specialized user in two days!Several participants spotted on the Grid later on!
NGIn School programNGIn School program
Day 1 (General Grid Introduction) Intro to Grid Intro to ARC First steps with ARC (tutorials)
Day 2 (Specialized tutorials) HEP distributed analysis ARC in bioinformatics, ARC in medical imaging ARC-gLite interoperability Grid Job Manager Dynamic Runtime Environments
Day 3 (ARC Development) Sys-admin and developer training
Next Grid SchoolNext Grid School
Next Grid School planned to be held during NorduGrid2008 in Budapest, Fall 2008
Similar program as in the first school
See you there!
SummarySummary
20/04/23 www.nordugrid.org 24
NorduGrid is a collaboration developing and deploying the ARC middlewareNDGF provides infrastructure and deploys and contributes to the current ARCKnowARC develops the next generation ARC NGIn educates and trains new Grid experts, both through PhD projects and Grid schoolsThe PhD projects contributes to both current ARC and the next generation ARCNext Grid school in Budapest, Fall 2008
20/04/23 www.nordugrid.org 25
NorduGrid CollaborationNorduGrid Collaboration
Official Address: NorduGrid,University of Oslo, Department of Physics Post Office box 1048, Blindern 0316 Oslo, NORWAY Tel: +47 2285 5056, Fax: +47 2285 6422
Steering Board:– Prof. F. Ould-Saada (Chair), Oslo– Prof. P. Eerola, Lund– Prof. T. Eckelöf, Uppsala– Dr. Miika Tsuiku, Helsinki– Dr. A. Wänänen, Copenhagen– New members (MoU to be extended):
• Dr. Pekka Lehtovuori, CSC, Finland• NN., NDGF• NN., KnowARC• Open to all ARC contributors
Technical contacts:– B. Konya, KnowARC– J. Kleist, NDGF– A. Read, NGIn
http://www.nordugrid.org/documents/NorduGrid-Agreement-05.pdf
http://www.nordugrid.org/about.html
20/04/23 www.nordugrid.org 26
Title of Contract: Grid-enabled Know-how Sharing Technology Based on ARC Services and Open Standards
Acronym: KnowARC Contract Nber: 032691 Instrument: STP - Specific Targeted Research Projects (aka STREP) Thematic priority/domain:
Information Society Technologies (IST) Call title: IST Call 5 Call identifier: FP6-2005-IST-5 Activity: IST-2005-2.5.4 - Advanced Grid Technologies, Systems and
Services Program: FP6 Duration: 36 month Start date:June 1, 2006 Community Contribution: EUR 2 899 494Resources:17.5 FTEs in total, 12.5 financedCoordinator: University of Oslo Contact: Prof. F. Ould-Saada@fys.uio.no , www.knowarc.eu
Official Project informationOfficial Project information