Tony [email protected]
“GridPP – Project Elements” UK e-Science All Hands Conference, Sheffield 3 September 2002
Tony Doyle - University of Glasgow
GridPP –GridPP – Project Elements Project Elements
• From Web to Grid…
• e-Science = Middleware
• LHC Computing Challenge
• Infrastructure– Tiered Computer Centres– Network
• BaBar – a running experiment
• Non-technical issues
• …Building the Next IT Revolution
• UK GridPP
• EU DataGrid – Middleware Development– Operational Grid
• DataGrid Testbed
• Status: 25 Jun 2002 16:38:47 GMT
• GridPP Testbed
• Grid Job Submission
• Things Missing, Apparently…
• …From Grid to Web
Tony Doyle - University of Glasgow
GridPP GridPP
EDG - UK Contributions
ArchitectureTestbed-1Network MonitoringCertificates & SecurityStorage Element R-GMALCFGMDS deploymentGridSiteSlashGridSpitfire…
Applications (start-up phase)
BaBarCDF/D0 (SAM)ATLAS/LHCbCMS(ALICE)UKQCD
£17m 3-year project funded by PPARC
CERN - LCG (start-up phase)
funding for staff and hardware...
£3.78m
£5.67m
£3.66m
£1.99m
£1.88m
CERN
DataGrid
Tier - 1/A
Applications
Operations
http://www.gridpp.ac.uk
Tony Doyle - University of Glasgow
Provide architecture and middleware
Use the Grid with simulated data
Use the Grid with real data
Future LHC Experiments
Running US Experiments
Build Tier-A/prototype Tier-1 and Tier-2 centres
in the UK and join worldwide effort to
develop middleware for the experiments
GridPP GridPP
Tony Doyle - University of Glasgow
Who are we?Who are we?
Nick White /O=Grid/O=UKHEP/OU=hepgrid.clrc.ac.uk/CN=Nick White member Roger Jones /O=Grid/O=UKHEP/OU=lancs.ac.uk/CN=Roger Jones member Sabah Salih /O=Grid/O=UKHEP/OU=hep.man.ac.uk/CN=Sabah Salih member Santanu Das /O=Grid/O=UKHEP/OU=hep.phy.cam.ac.uk/CN=Santanu Das member Tony Cass /O=Grid/O=CERN/OU=cern.ch/CN=Tony Cass member David Kelsey /O=Grid/O=UKHEP/OU=pp.rl.ac.uk/CN=David Kelsey member
Henry Nebrensky /O=Grid/O=UKHEP/OU=brunel.ac.uk/CN=Henry Nebrensky member Paul Kyberd /O=Grid/O=UKHEP/OU=brunel.ac.uk/CN=Paul Kyberd member Peter Hobson /O=Grid/O=UKHEP/OU=brunel.ac.uk/CN=Peter R Hobson member Robin Middleton /O=Grid/O=UKHEP/OU=pp.rl.ac.uk
/CN=Robin Middleton member Alexander Holt /O=Grid/O=UKHEP/OU=ph.ed.ac.uk/CN=Alexander Holt member Alasdair Earl /O=Grid/O=UKHEP/OU=ph.ed.ac.uk/CN=Alasdair Earl member Akram Khan /O=Grid/O=UKHEP/OU=ph.ed.ac.uk/CN=Akram Khan member Stephen Burke
/O=Grid/O=UKHEP/OU=pp.rl.ac.uk/CN=Stephen Burke member Paul Millar /O=Grid/O=UKHEP/OU=ph.gla.ac.uk/CN=Paul Millar member Andy Parker /O=Grid/O=UKHEP/OU=hep.phy.cam.ac.uk/CN=M.A.Parker member Neville Harnew /O=Grid/O=UKHEP/OU=physics.ox.ac.uk/CN=Neville Harnew member Pete Watkins /O=Grid/O=UKHEP/OU=ph.bham.ac.uk/CN=Peter Watkins member Owen Maroney /O=Grid/O=UKHEP/OU=phy.bris.ac.uk
/CN=Owen Maroney member Alex Finch /O=Grid/O=UKHEP/OU=lancs.ac.uk/CN=Alex Finch member Antony Wilson /O=Grid/O=UKHEP/OU=pp.rl.ac.uk/CN=Antony Wilson member Tim Folkes /O=Grid/O=UKHEP/OU=hepgrid.clrc.ac.uk/CN=Tim Folkes member Stan Thompson /O=Grid/O=UKHEP/OU=ph.
gla.ac.uk/CN=A. Stan Thompson member Mark Hayes /O=Grid/O=UKHEP/OU=amtp.cam.ac.uk/CN=Mark Hayes member Todd Huffman /O=Grid/O=UKHEP/OU=physics.ox.ac.uk/CN=B. Todd Huffman member Glenn Patrick /O=Grid/O=UKHEP/OU=pp.rl.ac.uk/CN=G N Patrick member Pete
Gronbech /O=Grid/O=UKHEP/OU=physics.ox.ac.uk/CN=Pete Gronbech member Nick Brook /O=Grid/O=UKHEP/OU=phy.bris.ac.uk/CN=Nick Brook member Marc Kelly /O=Grid/O=UKHEP/OU=phy.bris.ac.uk/CN=Marc Kelly member Dave Newbold /O=Grid/O=UKHEP/OU=phy.bris.ac.uk/CN=Dave
Newbold member Kate Mackay /O=Grid/O=UKHEP/OU=phy.bris.ac.uk/CN=Catherine Mackay member Girish Patel /O=Grid/O=UKHEP/OU=ph.liv.ac.uk/CN=Girish D. Patel member David Martin /O=Grid/O=UKHEP/OU=ph.gla.ac.uk/CN=David J. Martin member Peter Faulkner /O=Grid/O=UKHEP/OU=ph.
bham.ac.uk/CN=Peter Faulkner member David Smith /O=Grid/O=UKHEP/OU=ph.bham.ac.uk/CN=David Smith member Steve Traylen /O=Grid/O=UKHEP/OU=hepgrid.clrc.ac.uk/CN=Steve Traylen member Ruth Dixon del Tufo /O=Grid/O=UKHEP/OU=hepgrid.clrc.ac.uk/CN=Ruth Dixon del
Tufo member Linda Cornwall /O=Grid/O=UKHEP/OU=hepgrid.clrc.ac.uk/CN=Linda Cornwall member /O=Grid/O=UKHEP/OU=hep.ucl.ac.uk/CN=Yee-Ting Li member Paul D. Mealor /O=Grid/O=UKHEP/OU=hep.ucl.ac.uk/CN=Paul D Mealor member /O=Grid/O=UKHEP/OU=hep.ucl.ac.uk
/CN=Paul A Crosby member David Waters /O=Grid/O=UKHEP/OU=hep.ucl.ac.uk/CN=David Waters member Bob Cranfield /O=Grid/O=UKHEP/OU=hep.ucl.ac.uk/CN=Bob Cranfield member Ben West /O=Grid/O=UKHEP/OU=hep.ucl.ac.uk/CN=Ben West member Rod Walker /O=Grid/O=UKHEP/OU=hep.ph.
ic.ac.uk/CN=Rod Walker member /O=Grid/O=UKHEP/OU=hep.ph.ic.ac.uk/CN=Philip Lewis member Dave Colling /O=Grid/O=UKHEP/OU=hep.ph.ic.ac.uk/CN=Dr D J Colling member Alex Howard /O=Grid/O=UKHEP/OU=hep.ph.ic.ac.uk/CN=Alex Howard member Roger
Barlow /O=Grid/O=UKHEP/OU=hep.man.ac.uk/CN=Roger Barlow member Joe Foster /O=Grid/O=UKHEP/OU=hep.man.ac.uk/CN=Joe Foster member Alessandra Forti /O=Grid/O=UKHEP/OU=hep.man.ac.uk/CN=Alessandra Forti member Peter Clarke /O=Grid/O=UKHEP/OU=hep.ucl.ac.uk/CN=Peter
Clarke member Andrew Sansum /O=Grid/O=UKHEP/OU=hepgrid.clrc.ac.uk/CN=Andrew Sansum member John Gordon /O=Grid/O=UKHEP/OU=hepgrid.clrc.ac.uk/CN=John Gordon member Andrew McNab /O=Grid/O=UKHEP/OU=hep.man.ac.uk/CN=Andrew McNab member
Richard Hughes-Jones /O=Grid/O=UKHEP/OU=hep.man.ac.uk/CN=Richard Hughes-Jones member Gavin McCance /O=Grid/O=UKHEP/OU=ph.gla.ac.uk/CN=Gavin McCance member Tony Doyle /O=Grid/O=UKHEP/OU=ph.gla.ac.uk/CN=Tony Doyle admin Alex Martin /O=Grid/O=UKHEP/OU=ph.qmw.ac.uk/CN=A.J.Martin member Steve Lloyd /O=Grid/O=UKHEP/OU=ph.qmw.ac.uk/CN=S.L.Lloyd admin John Gordon
/O=Grid/O=UKHEP/OU=hepgrid.clrc.ac.uk/CN=John Gordon member
Tony Doyle - University of Glasgow
GridPP VisionGridPP Vision
From Web to Grid - Building the next IT Revolution
PremiseThe next IT revolution will be the Grid. The Grid is a practical solution to the data-intensive problems that must be overcome if the computing needs of many scientific communities and industry are to be fulfilled over the next decade.
Aim
The GridPP Collaboration aims to develop and deploy a large-scale science Grid in the UK for use by the worldwide particle physics community.
Many Challenges..Shared distributed
infrastructure For all applications
Tony Doyle - University of Glasgow
GridPP ObjectivesGridPP Objectives
1. SCALE: GridPP will deploy open source Grid software (middleware) and hardware infrastructure to enable the testing of a prototype of the Grid for the LHC of significant scale.
2. INTEGRATION: The GridPP project is designed to integrate with the existing Particle Physics programme within the UK, thus enabling early deployment and full testing of Grid technology and efficient use of limited resources.
3. DISSEMINATION: The project will disseminate the GridPP deliverables in the multi-disciplinary e-science environment and will seek to build collaborations with emerging non-PPARC Grid activities both nationally and internationally.
4. UK PHYSICS ANALYSES (LHC): The main aim is to provide a computing environment for the UK Particle Physics Community capable of meeting the challenges posed by the unprecedented data requirements of the LHC experiments.
5. UK PHYSICS ANALYSES (OTHER): The process of creating and testing the computing environment for the LHC will naturally provide for the needs of the current generation of highly data intensive Particle Physics experiments: these will provide a live test environment for GridPP research and development.
6. DATAGRID: Open source Grid technology is the framework used to develop this capability. Key components will be developed as part of the EU DataGrid project and elsewhere.
7. LHC COMPUTING GRID: The collaboration builds on the strong computing traditions of the UK at CERN. The CERN working groups will make a major contribution to the LCG research and development programme.
8. INTEROPERABILITY: The proposal is also integrated with developments from elsewhere in order to ensure the development of a common set of principles, protocols and standards that can support a wide range of applications.
9. INFRASTRUCTURE: Provision is made for facilities at CERN (Tier-0), RAL (Tier-1) and use of up to four Regional Centres (Tier-2).
10. OTHER FUNDING: These centres will provide a focus for dissemination to the academic and commercial sector and are expected to attract funds from elsewhere such that the full programme can be realised.
Tony Doyle - University of Glasgow
GridPP Project Map GridPP Project Map - Elements- Elements
Tony Doyle - University of Glasgow
Rare Phenomena –Rare Phenomena –Huge BackgroundHuge Background
9 or
ders
of
mag
nitu
de!
The HIGGS
All interactions
Tony Doyle - University of Glasgow
LHC Computing ChallengeLHC Computing Challenge
Tier2 Centre ~1 TIPS
Online System
Offline Farm~20 TIPS
CERN Computer Centre >20 TIPS
RAL Regional Centre
US Regional Centre
French Regional Centre
Italian Regional Centre
InstituteInstituteInstituteInstitute ~0.25TIPS
Workstations
~100 MBytes/sec
~100 MBytes/sec
100 - 1000 Mbits/sec
•One bunch crossing per 25 ns
•100 triggers per second
•Each event is ~1 Mbyte
Physicists work on analysis “channels”
Each institute has ~10 physicists working on one or more channels
Data for these channels should be cached by the institute server
Physics data cache
~PBytes/sec
~ Gbits/sec or Air Freight
Tier2 Centre ~1 TIPS
Tier2 Centre ~1 TIPS
~Gbits/sec
Tier Tier 00
Tier Tier 11
Tier Tier 33
Tier Tier 44
1 TIPS = 25,000 SpecInt95
PC (1999) = ~15 SpecInt95
ScotGRID++ ~1 TIPS
Tier Tier 22
Tony Doyle - University of Glasgow
Tier-0 - CERNTier-0 - CERN
Commodity Processors +IBM (mirrored) EIDE
Disks..
Tony Doyle -University of Glasgow
LHC Computing ChallengeLHC Computing Challenge
Tier2 Centre ~1 TIPS
Online System
Offline Farm~20 TIPS
CERN Computer Centre>20 TIPS
RAL Regional Centre
US Regional Centre
French Regional Centre
Italian Regional Centre
InstituteInstituteInstituteInstitute ~0.25TIPS
Workstations
~100MBytes/sec
~100MBytes/sec
100 -1000Mbits/sec
•One bunch crossing per 25 ns
•100 triggers per second
•Each event is ~1 Mbyte
Physicists work on analysis “channels”
Each institute has ~10 physicists working on one or more channels
Data for these channels should be cached by the institute server
Physics data cache
~PBytes/sec
~ Gbits/sec or Air Freight
Tier2 Centre ~1 TIPS
Tier2 Centre ~1 TIPS
~Gbits/sec
Tier Tier 00
Tier Tier 11
Tier Tier 33
Tier Tier 44
1 TIPS = 25,000 SpecInt95
PC (1999) = ~15 SpecInt95
ScotGRID++ ~1 TIPS
Tier Tier 22
2004 Scale: ~1,000 CPUs~5 PBytes
Compute Element (CE)
Storage Element (SE)
User Interface (UI)
Information Node (IN)
Storage Systems..
2002 200520042003
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4Q1 Q2 Q3 Q4Q1 Q2 Q3 Q4
Prototype of Hybrid Event Store (Persistency Framework)
Hybrid Event Store available for general users
Distributed production using grid services
First Global Grid Service (LCG-1) available
Distributed end-user interactive analysis
Full Persistency Framework
LCG-1 reliability and performance targets
“50% prototype” (LCG-3) available
LHC Global Grid TDR
applicationsapplications
gridgrid
2002 200520042003
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4Q1 Q2 Q3 Q4Q1 Q2 Q3 Q4
2002 200520042003
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4Q1 Q2 Q3 Q4Q1 Q2 Q3 Q4
Prototype of Hybrid Event Store (Persistency Framework)
Hybrid Event Store available for general users
Distributed production using grid services
First Global Grid Service (LCG-1) available
Distributed end-user interactive analysis
Full Persistency Framework
LCG-1 reliability and performance targets
“50% prototype” (LCG-3) available
LHC Global Grid TDR
applicationsapplications
gridgrid
Tony Doyle - University of Glasgow
UK Tier-1 RALUK Tier-1 RAL
New Computing Farm
4 racks holding 156 dual 1.4GHz Pentium III cpus. Each box has 1GB of memory, a 40GB internal disk and 100Mb ethernet.
50TByte disk-based Mass Storage Unit
after RAID 5 overhead. PCs are clustered on network switches with up to 8x1000Mb ethernet out of each rack.
Tape Robotupgraded last yearuses 60GB STK 9940 tapes 45TB currrent capacitycould hold 330TB.
2004 Scale: 1000 CPUs0.5 PBytes
Tony Doyle -University of Glasgow
LHC Computing ChallengeLHC Computing Challenge
Tier2 Centre ~1 TIPS
Online System
Offline Farm~20 TIPS
CERN Computer Centre>20 TIPS
RAL Regional Centre
US Regional Centre
French Regional Centre
Italian Regional Centre
InstituteInstituteInstituteInstitute ~0.25TIPS
Workstations
~100MBytes/sec
~100MBytes/sec
100 -1000Mbits/sec
•One bunch crossing per 25 ns
•100 triggers per second
•Each event is ~1 Mbyte
Physicists work on analysis “channels”
Each institute has ~10 physicists working on one or more channels
Data for these channels should be cached by the institute server
Physics data cache
~PBytes/sec
~ Gbits/sec or Air Freight
Tier2 Centre ~1 TIPS
Tier2 Centre ~1 TIPS
~Gbits/sec
Tier Tier 00
Tier Tier 11
Tier Tier 33
Tier Tier 44
1 TIPS = 25,000 SpecInt95
PC (1999) = ~15 SpecInt95
ScotGRID++ ~1 TIPS
Tier Tier 22
Tony Doyle - University of Glasgow
UK Tier-2 ScotGRIDUK Tier-2 ScotGRID
ScotGrid Processing nodes at Glasgow 59 IBM X Series 330 dual 1 GHz Pentium III with 2GB memory • 2 IBM X Series 340 dual 1 GHz Pentium III with 2GB memory and dual ethernet • 3 IBM X Series 340 dual 1 GHz Pentium III with 2GB memory and 100 + 1000 Mbit/s ethernet • 1TB disk • LTO/Ultrium Tape Library • Cisco ethernet switches
ScotGrid Storage at Edinburgh• IBM X Series 370 PIII Xeon with 512 MB memory 32 x 512 MB RAM • 70 x 73.4 GB IBM FC Hot-Swap HDD
CDF equipment at Glasgow• 8 x 700 MHz Xeon IBM xSeries 370 4 GB memory 1 TB disk
Griddev testrig at Glasgow• 4 x 233 MHz Pentium II
2004 Scale: 300 CPUs0.1 PBytes
BaBar UltraGrid System at Edinburgh• 4 UltraSparc 80 machines in a rack 450 MHz CPUs in each 4Mb cache, 1 GB memory • Fast Ethernet and Myrinet switching
Tony Doyle -University of Glasgow
LHC Computing ChallengeLHC Computing Challenge
Tier2 Centre ~1 TIPS
Online System
Offline Farm~20 TIPS
CERN Computer Centre>20 TIPS
RAL Regional Centre
US Regional Centre
French Regional Centre
Italian Regional Centre
InstituteInstituteInstituteInstitute ~0.25TIPS
Workstations
~100MBytes/sec
~100MBytes/sec
100 -1000Mbits/sec
•One bunch crossing per 25 ns
•100 triggers per second
•Each event is ~1 Mbyte
Physicists work on analysis “channels”
Each institute has ~10 physicists working on one or more channels
Data for these channels should be cached by the institute server
Physics data cache
~PBytes/sec
~ Gbits/sec or Air Freight
Tier2 Centre ~1 TIPS
Tier2 Centre ~1 TIPS
~Gbits/sec
Tier Tier 00
Tier Tier 11
Tier Tier 33
Tier Tier 44
1 TIPS = 25,000 SpecInt95
PC (1999) = ~15 SpecInt95
ScotGRID++ ~1 TIPS
Tier Tier 22
Tony Doyle - University of Glasgow
NetworkNetwork
• Internal networking is currently a hybrid of – 100Mb(ps) to nodes of cpu farms – 1Gb to disk servers– 1Gb to tape servers
• UK: academic network SuperJANET4 – 2.5Gb backbone upgrading to 20Gb in 2003
• EU: SJ4 has 2.5Gb interconnect to Geant• US: New 2.5Gb link to ESnet and Abilene for researchers
• UK involved in networking development
– internal with Cisco on QoS– external with DataTAG
Tony Doyle - University of Glasgow
Grid issues – CoordinationGrid issues – Coordination
• Technical part is not the only problem • Sociological problems? resource sharing
– Short-term productivity loss but long-term gain
• Key? communication/coordination between people/centres/countries– This kind of world-wide close coordination across multi-national
collaborations has never been done in the past
• We need mechanisms here to make sure that all centres are part of a global planning
– In spite of different conditions of funding, internal planning, timescales etc
• The Grid organisation mechanisms should be complementary and not parallel or conflicting to existing experiment organisation
– LCG-DataGRID-eSC-GridPP– BaBar-CDF-D0-ALICE-ATLAS-CMS-LHCb-UKQCD
• Local Perspective: build upon existing strong PP links in the UK to build a single Grid for all experiments
Tony Doyle - University of Glasgow
Experiment Deployment Experiment Deployment
Tony Doyle - University of Glasgow
DataGrid Middleware DataGrid Middleware Work PackagesWork Packages
• Collect requirements for
middleware– Take into account requirements from application
groups
• Survey current technology– For all middleware
• Core Services testbed– Testbed 0: Globus (no EDG middleware)
• First Grid testbed release
• Testbed 1: first release of
EDG middleware
• WP1: workload– Job resource specification &
scheduling
• WP2: data management– Data access, migration & replication
• WP3: grid monitoring services– Monitoring infrastructure, directories
& presentation tools
• WP4: fabric management– Framework for fabric configuration
management & automatic sw installation
• WP5: mass storage management– Common interface for Mass Storage
Sys.
• WP7: network services– Network services and monitoring
Tony Doyle - University of Glasgow
DataGrid ArchitectureDataGrid Architecture
Collective ServicesCollective Services
Information & MonitoringInformation
& MonitoringReplica ManagerReplica Manager
Grid Scheduler
Grid Scheduler
Local ApplicationLocal Application Local DatabaseLocal Database
Underlying Grid ServicesUnderlying Grid Services
Computing Element Services
Computing Element Services
Authorization Authentication
and Accounting
Authorization Authentication
and Accounting
Replica CatalogReplica Catalog
Storage Element Services
Storage Element Services
SQL Database Services
SQL Database Services
Fabric servicesFabric services
ConfigurationManagement
ConfigurationManagement
Node Installation &Management
Node Installation &Management
Monitoringand
Fault Tolerance
Monitoringand
Fault Tolerance
Resource Management
Resource Management
Fabric StorageManagement
Fabric StorageManagement
Grid
Fabric
Local Computing
Grid Grid Application LayerGrid Application Layer
Data Management
Data Management
Job Management
Job Management
Metadata Management
Metadata Management
Object to File Mapping
Object to File Mapping
Service Index
Service Index
Tony Doyle - University of Glasgow
Authentication/AuthorizationAuthentication/Authorization
• Authentication (CA Working Group)– 11 national certification authorities– policies & procedures mutual trust– users identified by CA’s certificates
• Authorization (Authorization Working Group)– Based on Virtual Organizations (VO).– Management tools for LDAP-based membership lists.– 6+1 Virtual Organizations
VO’s
ALICE Earth Obs.
ATLAS Biomedical
CMS
LHCb Guidelines
CA’s
CERN
CESNET
CNRS
DataGrid-ES
GridPP
Grid-Ireland
INFN
LIP
NIKHEF
NorduGrid
Russian DataGrid
Tony Doyle - University of Glasgow
WP7 - EDG AuthorisationWP7 - EDG Authorisationgrid-mapfilegrid-mapfile generation generation
o=testbed,dc=eu-datagrid, dc=org
CN=Franz Elmer
ou=People
CN=John Smith
mkgridmap
grid-mapfile
VOVODirectoryDirectory
““AuthorizatioAuthorizationn
Directory”Directory”
CN=Mario Rossi
o=xyz,dc=eu-datagrid, dc=org
CN=Franz ElmerCN=John Smith
Authentication
Certificate
Authentication
Certificate
Authentication
Certificate
ou=People ou=Testbed1
ou=???
local users ban list
Tony Doyle - University of Glasgow
Current User Base Current User Base Grid Support CentreGrid Support Centre
• GridPP (UKHEP) CA uses primitive technology– It works but takes effort– 201 personal certs issued– 119 other certs issued
• GSC will run a CA for UK escience CA– Uses openCA; Registration Authority uses web
– We plan to use itWe plan to use it– Namespace identifies RA, not Project– Authentication not Authorisation
• Through GSC we have access to skills of CLRC eSC
• Use helpdesk to formalise support later in the rollout
UK e-ScienceUK e-Science
CertificationCertification
AuthorityAuthority
Scale Scale
Tony Doyle - University of Glasgow
EDG TestBed 1 StatusEDG TestBed 1 Status30 Aug 2002 17:3830 Aug 2002 17:38
Web interface showing status of (~400) servers at testbed 1 sites
Production Centres
Tony Doyle - University of Glasgow
GridPP Context (Externally)GridPP Context (Externally) Neil Neil GeddesGeddes
InteroperabilityInteroperability
Tony Doyle - University of Glasgow
InteroperabilityInteroperability
Trust RelationshipsTrust Relationships
Tony Doyle - University of Glasgow
GridPP Sites in Testbed(s)GridPP Sites in Testbed(s)
Tony Doyle - University of Glasgow
GridPP Sites in Testbed: GridPP Sites in Testbed: Status 30 Aug 2002 17:38 Status 30 Aug 2002 17:38
Tony Doyle - University of Glasgow
t0
t1
From Grid to Web…From Grid to Web…using GridSiteusing GridSite
Tony Doyle - University of Glasgow
DocumentationDocumentation
• GridPP Web Site:http://www.gridpp.ac.uk/
• EDG User Guide:http://marianne.in2p3.fr/datagrid/documentation/EDG-Users-Guide.html
• EDG User Guide: A biomedical user point of view.http://www.creatis.insa-lyon.fr/~johan/wp10/testbed1-userguide/
• JDL Howto:http://server11.infn.it/workload-grid/docs/DataGrid-01-TEN-0102-0_1-Document.pdf
• GDMP Guide:http://cmsdoc.cern.ch/cms/grid/userguide/userguide-gdmp-3.0.pdf
Tony Doyle - University of Glasgow
Job SubmissionJob Submission
1. Authenticationgrid-proxy-init
2. Job submission to DataGriddg-job-submit
3. Monitoring and controldg-job-statusdg-job-canceldg-job-get-output
4. Data publication and replicationglobus-url-copy, GDMP
5. Resource scheduling
JDL, sandboxes, storage elements
Linux text interfaces
implementedGUIs next..
Tony Doyle - University of Glasgow
Job Submission ExampleJob Submission Example
dg-job-submit /home/evh/sicb/sicb/bbincl1600061.jdl -o /home/evh/logsub/bbincl1600061.jdl:#Executable = "script_prod";Arguments = "1600061,v235r4dst,v233r2";StdOutput = "file1600061.output";StdError = "file1600061.err";InputSandbox = {"/home/evhtbed/scripts/x509up_u149","/home/evhtbed/sicb/mcsend","/home/evhtbed/sicb/fsize","/home/evhtbed/sicb/cdispose.class","/home/evhtbed/v235r4dst.tar.gz","/home/evhtbed/sicb/sicb/bbincl1600061.sh","/home/evhtbed/script_prod","/home/evhtbed/sicb/sicb1600061.dat","/home/evhtbed/sicb/sicb1600062.dat","/home/evhtbed/sicb/sicb1600063.dat","/home/evhtbed/v233r2.tar.gz"};OutputSandbox = {"job1600061.txt","D1600063","file1600061.output","file1600061.err","job1600062.txt","job1600063.txt"};
Tony Doyle - University of Glasgow
GUI - todayGUI - today
Tony Doyle - University of Glasgow
GUI Future?GUI Future?Web Services Web Services
Access via Grid CertificateAccess via Grid Certificate
Tony Doyle - University of Glasgow
GridPP –GridPP – Achievements and Issues Achievements and Issues
• 1st Year Achievements• Complete Project Map
– Applications: Middleware: Hardware
• Fully integrated with EU DataGrid and LCG Projects
• Rapid middleware deployment /testing
• Integrated US-EU applications development e.g. BaBar+EDG
• Roll-out document for all sites in the UK (Core Sites, Friendly Testers, User Only).
• Testbed up and running at 15 sites in the UK
• Tier-1 Deployment• 200 GridPP Certificates issued• First significant use of Grid by an
external user (LISA simulations) in May 2002
• Web page development (GridSite)
• Issues for Year 2• Status: 19 Jul 2002 17:52 GMT –
keep monitoring and improve testbed deployment efficiency
• Importance of EU-wide development of middleware
• Integrated Testbed for use/testing by all applications
• Reduce “integration” layer between middleware and application software
• Integrated US-EU applications development
• Tier-1 Grid Production Mode• Tier-2 Definitions and Deployment• Integrated Tier-1 + Tier-2 Testbed• Transfer to UK e-Science CA• Integration with other UK projects
e.g. AstroGrid, MyGrid…
Tony Doyle - University of Glasgow
GridPP Sites in Testbed: GridPP Sites in Testbed: Status 19 Jul 2002 17:52Status 19 Jul 2002 17:52
Project MapProject MapSoftware releases Software releases at each siteat each site
Tony Doyle - University of Glasgow
GridPP –GridPP – An Operational Grid An Operational Grid
• From Web to Grid…• Fit into UK e-Science
structures • LHC Computing – Particle
physicists will use experience in distributed computing to build and exploit the Grid
• Infrastructure – tiered computing down to the physicist desktop
• Importance of networking • Existing experiments have
immediate requirements• Non-technical issues =
recognising/defining roles (at various
levels)
• UK GridPP started 1/9/01• EU DataGrid • First Middleware ~1/9/01
Development requires a testbed with feedback– “Operational Grid”
• Status: 25 Jun 2002 16:38:47 GMT – a day in the life..
• GridPP Testbed is relatively small scale – migration plans reqd. e.g. for CA.
• Grid jobs are being submitted today.. user feedback loop is important..
• Grid tools web page development by a VO.
• Next stop. Web services…
Tony Doyle - University of Glasgow
SummarySummary
• A vision is only useful if its shared
• Grid success is fundamental for PP
1. Scale in UK? 0.5 Pbytes and 2,000 distrib. CPUs
GridPP in Sept 2004 2. Integration – ongoing.. 3. Dissemination – external
and internal4. LHC Analyses – ongoing
feedback mechanism..5. Other Analyses – closely
integrated using EDG tools
6. DataGrid - major investment = must be (and is so far) successful
7. LCG – Grid as a Service 8. Interoperability – sticky subject9. Infrastructure – Tier-A/1 in
place, Tier-2’s to follow… 10. Finances – (very well) under
control • Next steps on framework VI..• CERN = EU’s e-science centre?• Co-operation required with
other disciplines/industry esp. AstroGrid
Top Related