Introduction
Edinburgh - Tsukuba Workshop 2
The ideas of grid computing are everywhereeven if the actual grids are not as pervasive
22/02/2009
The are many grid middleware packages with overlapping functionality
No universal solutionEach project requires some glue to tie components together
Outline
• Three example projects at EPCC
• OGSA-DAI
• BEinGRID
• ILDG
22/02/2009 Edinburgh - Tsukuba Workshop 3
Challenges
• Diversity– Data resource types, vendors, middleware, schema, meta
data
• Scale– Collections, formats, volumes, geographical, political and
social distance
• Ownership– On individual, group, and organisational levels
• Security– Client, service and data owners
22/02/2009 4Edinburgh - Tsukuba Workshop
Sharing data
22/02/2009 5Edinburgh - Tsukuba Workshop
• Convert data into information• Reveal new insights
– Scientific knowledge– Business advantage
• Data mining across distributed data resources– Exploit public and private data
• Open or closed communities– Scientific collaborations– Business partnerships
OGSA-DAI
• OGSA-DAI – 02/2002 – 07/2003– EPCC, NeSC, IBM, Oracle, NEReSC, eSNW
– DTI/EPSRC via UK e-Science Grid Core Programme
• DAIT (DAI-Two) – 10/2003 – 10/2005– EPCC, NeSC, IBM, NEReSC, eSNW
– DTI/EPSRC via UK e-Science Grid Core Programme 2 as part of the OMII-UK project
• OMII-UK – 11/2005 – 04/2009– EPCC, NeSC
– EPSRC
• OMII-UK extension – 04/2009 – 04/2010– EPCC, NeSC
– EPSRC
22/02/2009 6Edinburgh - Tsukuba Workshop
Pays Capital
l'Espagne Madrid
l'Italie Rome
Pays Capital
Grande-Bretagne Londres
France Paris
Convert query from French to English
Convert query from French to English
Convert query from French to Spanish
Convert query from French to Spanish
Run SQL query
Run SQL query
Convert data from English to
French
Convert data from English to
French
Join the data
Join the data
País Capital
España Madrid
Italia Roma
Country Capital
UK London
France Paris
SELECT Country, Capital FROM Countries
SELECT País, Capital FROM Países
SELECT Pays,Capital FROM Pays
Run SQL query
Run SQL query
Convert data from
Spanish to French
Convert data from
Spanish to French
Pays Capital
Grande-Bretagne Londres
France Paris
l'Espagne Madrid
l'Italie Rome
Activity
Activity input
Activity outputTarget data resource
Workflows
22/02/2009 7Edinburgh - Tsukuba Workshop
• Advanced Data Mining and Integration Research for
Europe– EU 7th Framework program project– EPCC, NeSC and European partners
• Infrastructure for data integration and mining– Large scale enterprise systems
• Applications– Flood modelling and simulations– Customer relationship management
ADMIRE
22/02/2009 8Edinburgh - Tsukuba Workshop
GEOGrid
• Global Earth Organisation (GEO) Grid– National Institute of Advanced Industrial Science and Technology,
Japan
• Geo-spatial data and services– Disaster mitigation– Environmental monitoring– Natural resource exploration– Virtual integration and access control
• Data– Satellite imagery– Geological data– Ground-sensed data
22/02/2009 9Edinburgh - Tsukuba Workshop
GLS Portal
Deliver
Deliver
Transform
Transform
JoinJoinGetGet
GetGet
Maps
1: GLSQuery submited via
portal e.g. “Leeds population
distribution by census output
area”
4: URL of image is returned to portal – avoids costly SOAP/HTTP transfer of image
5: Portal gets image using URL
Image Creation Service
MIMASCensus
UK
BORDERS
OGSA-DAI
2: Workflow is populated with query parameters and run
3: Image is placed on a map
server
SEE-GEO – geo-linking portal
22/02/2009 10Edinburgh - Tsukuba Workshop
Edinburgh - Tsukuba Workshop 11
BEinGRID
• Type of project: Integrated Project
• Project coordinator: ATOS ORIGIN
• Project start date*: 1st June 2006
• Duration: 42 months
• Max EC contribution: 15.7 M euros
• Consortium: 99 partners
http://www.beingrid.eu/
http://www.it-tude.com/
22/02/2009
Edinburgh - Tsukuba Workshop 12
BEinGRID Vision
• Typical Technology Transfer project:– 2 waves of 18+7 Business Experiments involving:
– SMEs in various industry-sections– Technical and Business experts
– Set up a repository of Grid solutions, available free/at cost to the respective sectors
– Prove that businesses will benefit from the adoption of Grid technologies
22/02/2009
Edinburgh - Tsukuba Workshop 13
BE02 – FilmGrid
• “Movie post-production workflow”
• Reviewing data flow in the industry– Current data movement tied into celluloid shooting– What is the effect of digital capture?– How useful is Sohonet other than for email?
• The FilmGrid prototype proves:– Grid technology is highly appropriate for movie post-production– Potentially large gains in:
– Efficiency– Reliability– Accountability– Accessibility
• http://tinyurl.com/filmgrid
22/02/2009
Edinburgh - Tsukuba Workshop 14
Asset Manager
Global AssetsLocal Files Transfer Status
22/02/2009
Edinburgh - Tsukuba Workshop 15
Database Triggers
• Procedure to be executed when a modification is
made to a table– INSERT, UPDATE or DELETE
• Various use cases– Log changes– Execute business rules (e.g. email a
manager when online orders push stock
levels below a specified threshold )– Enforce business rules (e.g. all invoices
must be associated with a valid customer)
• How to set-up a trigger is dependent on DB implementation
22/02/2009
Edinburgh - Tsukuba Workshop 16
OGSA-DAI Trigger
• Uses database triggers to call an OGSA-DAI
workflow upon modification to a database
• Extends single-database trigger functionality to:– Span several, heterogeneous databases– Execute powerful OGSA-DAI workflows
• Many possible use cases– Synchronising databases– Logging to an external database– Ensuring or executing business logic across partners
http://tinyurl.com/ogsadaitrigger
22/02/2009
Edinburgh - Tsukuba Workshop 17
BE24 – GRID2(B2B)
• “Grid technologies for affordable data synchronization and SME integration within B2B networks”
• Empowering existing B2B networks by electronically connecting suppliers at an affordable price– Webservices-based add-on to allow data exchange at database
level– Uses OGSA-DAI Trigger to automate synchronization
• The GRID2(B2B) prototype demonstrates:– Easy integration with multiple B2B platforms– User in total control of what data is sent– Automated synchronization:
• Fast and frequent data transfer• Remove the need to enter data twice
• http://tinyurl.com/grid2b2b
22/02/2009
Edinburgh - Tsukuba Workshop 18
How does it work?
DBMS
DBMS
Ducati - Starter Bentivogli - Partner
MaNeM – B2B Platform
GRID2(B2B) Data Federation Agent
GRID2(B2B)Data Service
New orders generated by Ducati software
DBMS
Orders written to an internal database
GRID2(B2B)Data Service
OGSA-DAI Trigger used to monitor for new data
Data Service communicates the new information to the Data Federation Agent
Data Service and Data Federation Agent are configured using the GRID2(B2B) Configurator
Data Federation Agent inserts information into B2B database.
Data Federation Agent also monitors for new data in the B2B platform and propagates it on to the correct member of the network
22/02/2009
International Lattice Data Grid
22/02/2009 Edinburgh - Tsukuba Workshop 19
• Sharing Lattice QCD data
• ILDG has no formal role– groups collaborate informally– working groups for metadata and middleware
• Individual groups were already starting to build data grid
infrastructures– UKQCD – QCDgrid, later DiGS– German groups combined into LATFOR, grid arm is LDG– US groups formed USQCD– Japanese – JLDG– Australia – Web portal
• Middleware often dictated by national considerations– ILDG is an aggregation of existing grids– Interoperable
ILDG WG
• Edinburgh and Tsukuba personnel
• Metadata Working Group– Tomoteru Yoshie Previous Convener– Chris Maynard Current Convener
• Middleware Working Group– George Beckett, Daragh Byrne, Eilidh Grant, Radek Ostrowski, and
James Perry– Mitsuhisa Sato, Toshiyuki Amagassa, Osamu Tatebe
• Example of Tsukuba and Edinburgh active collaboration
22/02/2009 Edinburgh - Tsukuba Workshop 20
Three requisite conditions
• Trust– already established in the community
– known community
• Altruism– political will to make data available– effort to build infrastructure– effort actually making data available
• Reward– how to credit those making data available– data users should cite a designated paper
22/02/2009 Edinburgh - Tsukuba Workshop 21
Three ideas to make this work
• Standard data format– Doesn’t really matter what, as long as one can read and write– configurations: SciDAC LIME record is 3x3 NERSC data layout
• Standard metadata– Semantic description of the data – Can be processed by an application
• Standard interfaces to services– Queries to metadata catalogues (MDC)– Queries to File Catalogue Web services (FC)– Authentication and authorisation
22/02/2009 Edinburgh - Tsukuba Workshop 22
Architecture
22/02/2009 Edinburgh - Tsukuba Workshop 23
Summary
• Rise in data complexity– doing things by hand is no longer scalable– we need tools to automate logistics and glue systems and data
together
• Grid architecture sits on top of existing systems– can access remote data with local tools– Many different middleware stacks
– Effort required to ensure interoperability
• Tsukuba and Edinburgh already collaborated successfully on
ILDG
22/02/2009 Edinburgh - Tsukuba Workshop 24
Lunch
22/02/2009 Edinburgh - Tsukuba Workshop 25
Top Related