Databases and the Grid OGSA-DAI Architecture & Requirements Malcolm Atkinson OGSA-DAI Chief...
-
Upload
robert-riley -
Category
Documents
-
view
214 -
download
0
Transcript of Databases and the Grid OGSA-DAI Architecture & Requirements Malcolm Atkinson OGSA-DAI Chief...
Databases and the Grid
OGSA-DAIArchitecture & Requirements
Malcolm AtkinsonOGSA-DAI Chief Architect
Director of National e-Science Centrewww.nesc.ac.uk
30th May 2002OGSA Early Adopters’ Workshop
Argonne National Laboratories
Overview
UK e-ScienceScale, Coordination, Structure, Projects
Database Task Force & GGF DAI-WGOGSA-DAI Project
Scope, Scale, Participants, PlansArchitectureRelationship with OGSARequirements
£80m Collaborative projects
E-ScienceSteering
Committee
DG Research Councils
Director
Director’s Management Role
Director’sAwareness and Co-ordination Role
Generic Challenges EPSRC (£15m), DTI (£15m)
Industrial Collaboration (£40m)
Academic Application SupportProgramme
Research Councils (£74m), DTI (£5m)
PPARC (£26m) BBSRC (£8m) MRC (£8m) NERC (£7m) ESRC (£3m) EPSRC (£17m) CLRC (£5m)
Grid TAG
UK e-Science Programme
Tony Hey
Cambridge
Newcastle
Edinburgh
Oxford
Glasgow
Manchester
Cardiff
Southampton
London
Belfast
Daresbury Lab
RALHinxton
UK Grid Network
AccessGrid always-on video
walls
AccessGrid always-on video
walls
Nationale-
ScienceCentre
NeSC’s Roles
TAG
NeSC
eSIGSC
Application Pilots IRCs …e-Science Centrese-Scientists, Grid users, Grid services & Grid Developers
UK Core Directorate Global Grid Forum …
CS Research
DBTF ATFGNT
Coordination, Stimulation & Education
ETF
STF
UK Architectural Task Force (ATF)
Malcolm Atkinson (NeSC) Geof. Coulson (Lancaster U.)
Jon Crowcroft (Cambridge U.) David De Roure (Southampton U.)
Vijay Dialani (Southampton U.) Andrew Herbert (Microsoft)
Ian Leslie (Cambridge U.) Andrew Martin (Oxford U.)
Ken Moody (Cambridge U.) Steven Newhouse (ICSTM & LeSC)
Tony Storey (IBM)
…………… Plus consultations
UK Role in Open Grid Services Architecture, Version 0.6 11th March 2002
www.nesc.ac.uk
teams
ATFObtained Agreement: OGSA as Foundation for UK work, 18 April 2002
e-Science Institute
National e-Science Centre
Edinburgh + Glasgow Universities
Physics & Astronomy 2Informatics, Computing ScienceEPCC
£6M EPSRC/DTI + £2M SHEFC over 3 years
www.nesc.ac.uk
e-Science Institutevisitors, workshops, co-ordination, outreach
middleware development50 : 50 industry : academia
‘last-mile’ networking
e-Science Institutevisitors, workshops, co-ordination, outreach
middleware development50 : 50 industry : academia
‘last-mile’ networking
UK Pilot Projects
Research Councils Autonomy> 30 Projects$5 million to $0.3 million
Wide Range of DisciplinesIndustrial InvolvementIntegration and Access to Informatione-Science Centre Projects
> 50% Industrial Involvement
Equator: Technological innovation in physical and digital life
AKT: Advanced Knowledge Technologies
DIRC: Dependability of Computer-Based Systems
MIAS: From Medical Images and Signals to Clinical Information
IRC ‘Grand Challenge’ Projects IRC ‘Grand Challenge’ Projects
From presentation by Tony Hey
Particle Physics and Astronomy e-Science Projects
GridPPlinks to EU DataGrid, CERN LHC Computing Project, US GriPhyN and PPDataGrid Projects, and iVDGL Global Grid Project
AstroGridlinks to EU AVO and US NVO projects
From presentation by Tony Hey
OGSA-DAI Early Adopter
Comb-e-Chem:Structure-Property Mapping
Southampton, Bristol, Roche, Pfizer, IBM
DAME: Distributed Aircraft Maintenance Environment
York, Oxford, Sheffield, Leeds, Rolls Royce
Reality Grid: A Tool for Investigating Condensed Matter and Materials
QMW, Manchester, Edinburgh, IC, Loughborough, Oxford, Schlumberger, …
EPSRC e-Science Projects (1)
From presentation by Tony Hey
EPSRC e-Science Projects (2)
MyGrid: Personalised Extensible Environments for Data Intensive in silico Experiments in Biology
Manchester, EBI, Southampton, Nottingham, Newcastle, Sheffield, GSK, Astra-Zeneca, IBM, Sun
GEODISE: Grid Enabled Optimisation and Design Search for Engineering
Southampton, Oxford, Manchester, BAE, Rolls Royce
Discovery Net: High Throughput Sensing Applications
Imperial College, Infosense, …
From presentation by Tony Hey
OGSA-DAI Early Adopter
MyGrid e-Science Workbench
Goal is to develop ‘workbench’ to support:Experimental process of data accumulationUse of community informationScientific collaboration
Provide facilities for resource selection, data management and process enactmentBioinformatics applications
Functional genomics, pattern database annotation
Manchester, EBI, Newcastle,Nottingham, Sheffield, SouthamptonGSK, AstraZeneca, Merck, IBM, Sun, ...
From presentation by Tony Hey
Overview
UK e-ScienceScale, Coordination, Structure, Projects
Database Task Force & GGF DAI-WG
OGSA-DAI Project
Scope, Scale, Participants, PlansArchitectureRelationship with OGSARequirements
DBTF Web Pages
http://www.cs.man.ac.uk/grid-db
DBTF Membership
Malcolm Atkinson (NESC)Vijay Dialani (Southampton University)Norman Paton (Manchester University)Dave Pearson (Oracle UK)Tony Storey (IBM Hursley)Paul Watson (Newcastle University)
DBTF: Aims & Actions
Requirements Capture
Pilot Project MeetingsReportDave Pearson
RoadmapUK CoordinationGGF Articulation
StandardsBoF GGF4Papers GGF5
ImplementationProjectsOGSA-DAI
ArchitectureLiase with ATFLiase with Globus team
Educatione-Science InstitutePilot ProjectsGSC
EvolvingGGF DAIS WGBroader community
Overview
UK e-ScienceScale, Coordination, Structure, Projects
Database Task Force & GGF DAI-WG
OGSA-DAI Project Scope, Scale, Participants, PlansArchitectureRelationship with OGSARequirements
CambridgeOxford
Glasgow
Cardiff
Southampton
London
Belfast
Daresbury Lab
RALHinxton
OGSA-DAI Partners
EPCC & NeSC
Newcastle
IBMUSA
IBM Hurseley
Oracle
Manchester
EPCC & NeSCIBM UKIBM USAManchester e-SCNewcastle e-SCOracle $5 million, 18 months, started 1st February 2002
OGSA-DAI Scope
Definition and development of generic Grid data services which provide access to and integration of data held in databases, and the management of data within a distributed environment. Database
A stored, structured collection of dataAccessed using an API that takes account of the structure of the data stored
IncludesRelational and object databasesXML repositoriesAdequately described collections of files
Databases in the Grid
Computational Complexity
DataComplexity
Scope of Database Services
Discovery of Data by ContentQuery and Update StatementsMetadata Management & EvolutionTransactions (Flavours of)Distributed queries and updatesSpecialised typesEncapsulated (safe) Function applicationNotification (driven by triggers, etc.)
OGSA-DAI ObjectivesProduce specifications for generic data services
based on a common design framework consistent with Open Grid Service Architecture
Design specificationsas basis of standards recommendations via Database Access and Integration Services Working Group to the Global Grid Forum
Deliver Grid data services software in future releases of the Globus Toolkit (GT3 December 2002)
Refine identified requirementsevaluate design options develop demonstratorstransfer skills to the Grid community
Develop reference implementations of generic data servicesEnsure that the Grid model and OGSA standards address fully the needs of data
access and integration Ensure Grid data services meet the levels of service required
performance, scalability, resilience, availability, and manageabilityevolution and distribution
large user populations and large data volumes
OGSA-DAI Plan
Two PhasesPhase 1: Started Feb 02 ends GGF5Detailed Plan –
Requirements, Designs & Prototypes 6 Work Packages Project Management (Oracle, EPCC) Architecture (NeSC, DBTF) XML Data Management (NeSC & EPCC) Distributed Query Systems (Manchester & Newcastle) Metadata & Registries (NeSC & EPCC) Relational Databases (IBM UK)
Phase 2: 12 months Structure and Objectives to be Refined in Major Review GGF5 DAIS WG meeting a major input
OGSA-DAI Time Line
Feb ’02 May ’02 Jul ’02 Sep ’02 Dec ’02 Feb ’03 May ’03 Sep ’03
Ship for GT3 Integration
RDB + GT2 / OGSA Prototypes Available
XML + OGSA Prototype Available
Design Documents & Demos for DAIS WG @ GGF5
RDB + GT2 / OGSA Prototypes for Early Adopters
XML + OGSA Prototypes for Early Adopters
WS + GSI UK support ( > 60 downloads)
Phase 2 StartsPhase 1 Starts
Milestones & Deliverables3rd Jul 2002 GGF 5 Deliverables
1st Draft – OGSA-DAI Design SpecificationWorking Grid data service prototype with workshop materialDraft Phase 2 functional scope for each Work Package
30th Sept 2002
End Phase 1Phase 1 Review Report and recommendations including: revisions to Phase 2 streams of work, Work Package structure, content, and scopeCompleted, Tested, Work Package prototypes with evaluation report detailing functional scope and deficiencies, design options, measures for acceptanceRDBMS/Globus-2 prototype implementation Phase 2 scopeAgreed 2nd Draft – OGSA-DAI design specificationDissemination programme for UK e-Science communityTransition programme for UK Grid Support Team and Globus Development Team
31st Dec 2002
Globus Toolkit Release1st Grid data services reference implementation for Globus Toolkit 31st Grid data services specification for Globus Toolkit 3Scope of functional content for 2nd Globus Toolkit release and specification1st release training and support courses
31st Mar 2003
Interim UK e-Science community releaseInterim Grid data services implementation for UK e-Science communityRelease training and support courses, with documentation
31st Jul 2003
Globus Toolkit Release2nd Grid data services reference implementation for Globus Toolkit 32nd Grid data services specification for Globus Toolkit 32nd release training and support coursesPublications and papers to support reference implementations through WG discussions and GGF standards processesFinal Project Report
OGSA-DAI: Key Components
Grid Database Services (GDS)GXDS, GRDS, GSFDS, …Perform DB actionsExtra Data Service ElementsDB-action-Management FunctionsNotifications from Triggers
Grid Database Service Factories (GDSF)Create the aboveExtra Data Service Elements
Database Service Registries (DSR)Specialised Registries to find DBs, Services & Factories
Grid Data Transfer Services (GDTS)Described at Requirement LevelFlexible & mapped to grid-FTP, MQ Series, …
OGSA-DAI Architecture
1requestforfactory
DSR
GDSF
client
OGSA-DAI Architecture
2responsewithGDSFsGSHs
1requestforfactory
DSR
GDSF
client
OGSA-DAI Architecture
2responsewithGDSFsGSHs
1requestforfactory
3script for 3GDSs
DSR
GDSF
client
4creation of 3GDSs
OGSA-DAI Architecture
2responsewithGDSFsGSHs
1requestforfactory
3script for 3GDSs
DSR
GDSF
GDS1
GDS2
GDS3
client
4creation of 3GDSs
OGSA-DAI Architecture
5responsewith 3GSHs
2responsewithGDSFsGSHs
1requestforfactory
3script for 3GDSs
DSR
GDSF
GDS1
GDS2
GDS3
client
4creation of 3GDSs
OGSA-DAI Architecture
6scriptsrequestingDB actions
5responsewith 3GSHs
2responsewithGDSFsGSHs
1requestforfactory
3script for 3GDSs
DSR
GDSF
GDS1
GDS2
GDS3
client
4creation of 3GDSs
OGSA-DAI Architecture
6scriptsrequestingDB actions
5responsewith 3GSHs
2responsewithGDSFsGSHs
1requestforfactory
3script for 3GDSs
DSR
GDSF
GDS1
GDS2
GDS3
client
7transfer databatch to GDS2
stream to GDS3
4creation of 3GDSs
OGSA-DAI Architecture
6scriptsrequestingDB actions
5responsewith 3GSHs
2responsewithGDSFsGSHs
1requestforfactory
3script for 3GDSs
DSR
GDSF
GDS1
GDS2
GDS3
client
7transfer databatch to GDS2
stream to GDS3
8stream datato GDS2
4creation of 3GDSs
OGSA-DAI Architecture
6scriptsrequestingDB actions
5responsewith 3GSHs
2responsewithGDSFsGSHs
1requestforfactory
3script for 3GDSs
DSR
GDSF
GDS1
GDS2
GDS3
client
9transfer databatchto client
7transfer databatch to GDS2
stream to GDS3
8stream datato GDS2
4creation of 3GDSs
OGSA-DAI Architecture
6scriptsrequestingDB actions
5responsewith 3GSHs
2responsewithGDSFsGSHs
1requestforfactory
3script for 3GDSs
DSR
GDSF
GDS1
GDS2
GDS3
client
9transfer databatchto client
7transfer databatch to GDS2
stream to GDS3
8stream datato GDS2
10stream datato specifieddestination
OGSA-DAI & OGSA <((-:}Description, e.g. portType Works Well
Adding only one portType / GDS(F) | DSR
Expect to make extensive use ofData Service Elements
Special to DBs: Static & Dynamic
Component ManagementNotificationGrid-FTPAccountingSecurity:
Authentication, Authorisation & Privacy
Reliable invocation…
OGSA-DAI & OGSA <))-:}Lifetime Issues
Conditions for terminationControlled clean-up opportunityScope of StateEvolution
Notification IssuesRegistering & using same notification system
For DBs, e.g. triggers do we have to construct a dummy Service Data Element?
Type System IssuesStandards needed for wide range of types
Service Definition IssuesHow to create / obtain standard definitions for common services
OGSA-DAI Summary
On Schedule & Going WellExpect Contributions via DAIS-WG @ GGF5Expect Contributions to GT3 ReleasesEarly Days
Testing Architectural DesignUsing OGSAWorking with Early Adopter Pilot Projects
AstroGrid & MyGrid
Planned release of prototypes
Influence OGSA-DAI directionVia DAIS-WG