A new Architecture for OGSA-DAI
13
A new Architecture for OGSA-DAI Malcolm Atkinson, K. Karasavvas, M. Antonioletti, R. Baxter, A. Borley, N. Chue Hong, A. Hume, M. Jackson, A. Krause, S. Laws, N. Paton, J. M. Schopf, K. Tourlas and P. Watson 19 th September 2005
description
A new Architecture for OGSA-DAI. Malcolm Atkinson, K. Karasavvas, M. Antonioletti, R. Baxter, A. Borley, N. Chue Hong, A. Hume, M. Jackson, A. Krause, S. Laws, N. Paton, J. M. Schopf, K. Tourlas and P. Watson 19 th September 2005. Contents: What we talk about. Background Requirements - PowerPoint PPT Presentation
Transcript of A new Architecture for OGSA-DAI
1
A new Architecture for OGSA-DAI
Malcolm Atkinson, K. Karasavvas, M. Antonioletti,
R. Baxter, A. Borley, N. Chue Hong, A. Hume, M. Jackson, A. Krause, S. Laws, N. Paton, J. M.
Schopf, K. Tourlas and P. Watson
19th September 2005
2
Contents: What we talk about
Background
Requirements
Overview of new Architecture
Status and Future Work
Background
OGSA-DAIMiddleware to provide uniform extensible access to data resources
Relational, XML, Files and Indexed Files Translation and delivery activities
Old releases Globus Toolkit 3 - OGSI
New releases Axis – WS-I OMII – WS-I+ Globus Toolkit 4 – WSRF
Experience Previous assumption: web services are lightweight Factory pattern: making a home for session state Cannot rely on all data resources having sufficient functionality
Old Architecture• One GDS per client interaction• One Data Resource per GDS• No explicit Sessions• No explicit Transactions• Depend on data resources for concurrency management
International Collaboration & Use
USA:o Globus Allianceo IBM Corporationo caBIGo BIRNo Indiana University o GridSphereo GEONo LEADo MCSo NCSAo Secure Data Grido UNC
Japan:o AISTo BioGrido NAREGI
Europe:o CERNo DataMiningGrido GridMinero GridSphereo inteligrido N2Grido OntoGrido Provenanceo SIMDATo OMII-EU
UK:o OMIIo OMII-UKo NGSo NCeSSo NIeeSo AstroGrido BioSimGrido BRIDGESo CancerGrido ConvertGrido eDiaMonDo EDINAo First Group plco Fujitsu Labs Europeo GEDDMo GeneGrido Genomic Technology and Informaticso GOLDo Human Genetics Unito IBM UKo myGrido Oracle UK
China:o CASo ChinaGrido cnGrido INWAo OMII-China
Australia:o Curtin Business Schoolo INWA
TutorialsBoston CambridgeCERN ChicagoEdinburgh LondonSan Francisco SeattleSeoul SingaporeTokyo ISSGC 03 to 05
DIALOGUE workshopsColumbus, Edinburgh, Indiana, Vienna
Chicago, Manchester, San Diego
South Korea:o KISTI
China40%
United Kingdom15%
United States11%
Germany3%
Japan5%
Italy2%
France3%
Austria1%
Others20%
1485 registered users5250+ downloads
LEAD
GeneGrid
caBIG
BRIDGES
OGSA WebDB
FirstDIG ConvertGrid eDiaMoND
OGSA-DQP
Grid Miner
Meeting User Requirements
RequirementsWeb Services now long-running
Manage state lifetime within WSManage concurrency within WSManage resources within WS
Continued support of functionality
All previous tasksAll previous extensibilitySingle task & composition
Accommodate small and large tasks
Standard activitiesManageable and Configurable
Monitoring and controlPersistent & resilient to failures
Persistent operational stateRecovery
PerformanceStreamed pipelines of tasks
Concurrent sessions & transactions
Standard coordination protocolsEmbedded in request documentsMulti-request sessions
Extensible framework & Increased Power in Requests
Application-defined activitiesAdditional data sourcesExtensible client libraries
Standard interfaces – Preserve application investment
Client library stabilityActivity API stability
NamingMultiple Data Resources per Data ServiceResults, streams, sessions, transactions, …
New OGSA-DAI Architecture
Tx Request
Tx Response
External Standard Coordination
Execution Model – Workload Throttle & Session manager
TADD
Receiverthread To do
queue
Activesession
pool
Freethreadpool
Allocaterthread
Sessionexists?
Capacityexists?
Yes resume
no new session
Yes createsession
no wait for session to end
Execution Model – Execution Planning
TADD
Receiverthread To do
queue
Activesession
pool
Freethreadpool
Allocaterthread
Sessionthread
Resume Session
Check authentication
unchanged
Parse toform task DAG
Check authorisation
Update DIDtable
TaskDAG
Optimise taskDAGOptimised
executionplan
Start executioncoordination
thread
Executioncoordination
thread
Monitor executioncoordination
thread
Resume Session
TaskForm
Execution Model – Processing one Request
TADDOptimisedexecutionplan (oep)
Executioncoordination
thread
TaskForm
input
Set Tx mode
start
Create resultTADD(s)
TADD
result
Initiate 3rd party datacollection
Walk oep launching
activity threadsactivitythread
activitythread
activitythread
activitythread
Initiate 3rd party datadelivery
detect completions
& tidy up
Detect overall completionor failures
Tidy & closeTADD(s)
Tidy DIDs
hibernateor end Tx
hibernateor pool ECT
Data Streams
Status and Future Work
Release 7 – Partial move to new architecture
New DR extensibility pointNew activities, e.g. multi-source unionReconfigurable servicesConcurrency within data serviceSessionsParts of DQP
DQP Release 3Releases for Friends & Family
Transactions (tech. preview of phases)
Status and Future Work 2
Following releasesPerformance
Execution planning, concurrency, streamed pipelines, better data transfer
Functionality Completed Transactions Distributed queries Resilient multi-source unions Schema and data integration More data extraction from files & files in transactions
New extensibility & configuration Activity database Dynamically installing extension activities
Further information
The OGSA-DAI Project Site:http://www.ogsadai.org.uk
The DAIS-WG site:http://forge.gridforum.org/projects/dais-wg/
OGSA-DAI Users Mailing [email protected] discussion on grid DAI matters
Formal support for OGSA-DAI releaseshttp://www.ogsadai.org.uk/[email protected]
OGSA-DAI training courses
QuestionsPlease
