GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del...
-
Upload
brandon-dean -
Category
Documents
-
view
213 -
download
0
Transcript of GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del...
![Page 1: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/1.jpg)
GOCDBA repository for a worldwide
grid infrastructure
G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen
CHEP09, Prague, March 2009
![Page 2: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/2.jpg)
Outline• What is GOCDB
– Context and generalities– Purpose and history– GOCDB interactions with other EGEE tools
• GOCDB architecture– Components, services and interfaces– Development and procedures– Availability and failover
• Evolution– GOCDB in a distributed environment
![Page 3: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/3.jpg)
What is GOCDB? (1)Context and generalities
• What’s in a name?– GOCDB = Grid Operations Centre DataBase
• A central (static) information repository
• Key component of EGEE and WLCG, used as an authoritative data source
![Page 4: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/4.jpg)
What is GOCDB? (2)Context and generalities
• Stores information about (and links together):– Regions and countries– Sites, nodes and services– Users
• Consists of 2 parts:– A Database– A web portal (https://goc.gridops.org)
![Page 5: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/5.jpg)
What is GOCDB? (3)Handled data
• Administrative info– Contacts, names, groupings (region, country…)– Used for official procedures, reports, listings
• Resources and services– service endpoints and types– Used for monitoring and operations
• Maintenance plans– Used for monitoring and availability reports
![Page 6: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/6.jpg)
What is GOCDB? (4)History
Stone Age Static list of sites and their contacts
Iron Age simple MySQL DB + scripts
Medieval times MySQL DB + proper web interface
Industrial Revolution Oracle DB + user friendly web portal
Modern times Oracle DB + Web portal + web services
2003
2004
2005
2006
2007
2008
2009
![Page 7: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/7.jpg)
Repositories of Information
Accounting MonitoringTicket
Followup
Reporting Alarms
User Support
What is GOCDB? (5)interactions
GOCDB, Operations Portal
APEL, Accounting
Enforcement Portal
SAM, GStatOperationsDashboard
GridViewAccounting Portal
Site Fabric Monitoring
GGUS
![Page 8: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/8.jpg)
Architecture overview
• Web portal– Based on a Model-view-Controller (MVC)
principle– Coded in PHP, object fashion
• Database– hosted on Oracle 11g cluster– 3 schemas: Production, test, development
• Authentication – Based on X509 certificates– Role management
![Page 9: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/9.jpg)
![Page 10: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/10.jpg)
Services and interfaces
• GOCDB Programmatic interface (GOCDB-PI)– REST based interface over https– Business logic put in stored functions at DB
level– Use of Oracle XML DB– Different methods with different security levels
• SOAP web services under study
![Page 11: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/11.jpg)
Processes and procedures
• Code synchronisation– GOCDB repository in SVN
• Release procedure– RPM deployment– IP switch between production and test server
• Development cycle– Savannah task/bug tracker– Well established validation process
![Page 12: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/12.jpg)
Development workflow
Mail/ otherGGUS UKI
Helpdesk
SavannahWish List
USER
GOCDB Admin
Request
Request
Status=new
Does the request need discussion?
no
yes
Request
Status=accepted
Request
Status=refused
Is the request acceptable?
no
yes
Discuss request
Discuss request
GOCDB AG
OAT
Does it need further
validation?
noyesOfficial Development
List
![Page 13: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/13.jpg)
GOCDB Failover (1)
• Web portal– Master instance at RAL, UK– Replica instance at ITWM, Germany
• Database– Master instance at RAL, UK– Replica instance at CNAF, Italy– Local backup instance at RAL, UK
Oracle 11g
Oracle 10g
Oracle Xpress
![Page 14: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/14.jpg)
GOCDB Failover (2)
Master DB
Replica DB
Backup DB
Master web portal
GOCDB-PI
Replica web portal
conf GOCDB-PIconf
https://goc.gridops.org DNS alias
3rd party tools
End users operator
![Page 15: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/15.jpg)
GOCDB evolution
• Model evolution in EGEE– From central to distributed– Automate, simplify, standardize
• Distributing “the beast”– New architecture under development– Challenging work and short deadline
![Page 16: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/16.jpg)
Main ideas• Keep a central service, not necessarily a central DB
– There is a need for a central access point, but:– the fact that regional DB are distributed or not must not
be an issue
• Build a sustainable architecture that allows regionalisation but doesn’t force it– Not all regions are at the same level
• Propose an implementation where nothing exists, work with existing solutions otherwise– Some regions have their own solution and don’t want to
be forced to use another one
![Page 17: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/17.jpg)
Region 1
GOCDB regional instance
RAL
GOCDB Central instance
Region 3
High level architecture
Central portal
Region 1 local portal
Region 3 homemade
local DB
Region 3 local portal3rd party tools
WS interface
Query interface
Query interface
Data collector
End usersPublish (push)
Query (pull)
![Page 18: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/18.jpg)
Region 1 RAL
Region 3
One level down
GOCDB internal model (regions
1, 2 and 3)
GOCDB internal model (regions
1, 2 and 3)
Central portal
GOCDB Region 1
model
Region 1 local portal
Region 3 homemade
local DB
Region 3 local portal
Publisher
3rd party tools
WS interface
Query interface
Query interface
Data collector
End users
![Page 19: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/19.jpg)
Region 1 RAL
Region 3
Custom tables
Details and components
Region 1Region 1
Region 2Region 2
Central portal
Region 1
Region 3Region 3
Region 1 local portal
Region 3 homemade
local DB
Region 3 local portal
Publisher
3rd party tools
WS interface
Query interface
R1R1
R2R2
R3R3
R1R1
data metadata
Query interface
Data collector
Data processor
End users
![Page 20: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/20.jpg)
internal DB schema
- ID- Sitename- email
SITES-ID-hostname-IP_addr
NODES
- ID- path- start_date- end_date
DOWNTIMES
- ID- name- desc
SERVICES
- path-SiteID-NodeID-ServiceID
PATHS
-Type- Name-Database-Grid
objectTypes
-Type-objectID
objects
- parent_type-Child_type-Allowed
objectLinkTypes
-ID-Link_type-Parent_objectID-Child_objectID
objectLinks
-GridID-ObjectID
- Sitename- email
TABLE1
-GridID-ObjectID
- hostname- IP_addr
TABLE2
-GridID-ObjectID
- serv_name- desc
TABLE3
-GridID-ObjectID
- start_date- end_date
TABLE4
-GridID-ObjectID
- Sitename- email
TABLE1.2
-GridID-ObjectID
- hostname- IP_addr
TABLE2.2
-GridID-ObjectID
- serv_name- desc
TABLE3.2
-GridID-ObjectID
- start_date- end_date
TABLE4.2
-GridID-ObjectID
- Sitename- email
TABLE1.3
-GridID-ObjectID
- hostname- IP_addr
TABLE2.3
-GridID-ObjectID
- serv_name- desc
TABLE3.3
-GridID-ObjectID
- start_date- end_date
TABLE4.3
Current relational model Proposed object model
Collection1
Core tables (relationships)
- Physical Data Tables- Hard coded relationships and constraints
Data tables
Collection2 Collection3
![Page 21: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/21.jpg)
For more details…
• GOCDB homepage– http://www.grid-support.ac.uk/content/view/406/290/
• GOCDB4 development wiki– http://goc.grid.sinica.edu.tw/gocwiki/GOCDB4_development
• “A pseudo object database model and its applications on a highly complex distributed architecture”
– IARA/IEEE Conference on Advances in Databases (DBKDA 2009) March 1-6, 2009 - Cancun, Mexico
![Page 22: GOCDB A repository for a worldwide grid infrastructure G. Mathieu, A. Richards, J. Gordon, C. Del Cano Novales, P. Colclough, M. Viljoen CHEP09, Prague,](https://reader034.fdocuments.in/reader034/viewer/2022042822/56649e395503460f94b2b0e8/html5/thumbnails/22.jpg)
Thank you
• Questions ?