Data Grid: GRASP Mike Smorul. Grid Retrieval and Search Platform Based on concepts developed in the...
-
date post
21-Dec-2015 -
Category
Documents
-
view
215 -
download
0
Transcript of Data Grid: GRASP Mike Smorul. Grid Retrieval and Search Platform Based on concepts developed in the...
Grid Retrieval and Search Platform
Based on concepts developed in the Earth Science Data Interface (ESDI) developed at the UMIACS GLCF.
Provides a graphical interface into data grid holdings.
Access to entire GLCF holdings through the Storage Resource Broker(SRB)
ESDI Overview
Designed to allow for intuitive browsing and searching of large geospatial data sets.
Tightly integrated set of web, ftp, and file servers. (customized to GLCF)
Distributes over 7Tb of data per month Over 27,000 Landsat scenes and 13Tb of
data available for download
SRB Grid Testbed (original)
Modified the SRB to hold spatial data Contributed Informix port of the SRB (v3.2) Linked three ESIP sites Tested replication between
GMU and UMD. Remote registration at
UNH
UMD MCAT enabled srbmaster
GMU srbmaster UNH srbmaster
Informix
MODISTM
MSS(1)MSS(2)
UMD srbmasterwith dfs access
Lessons Learned
The SRB can easily handle textual metadata.
Spatial data could be stored into extended Informix attributes, but querying was available only through the DAILimited SRB MCAT to Informix based
systems
GRASP Architecture
I/O Abstraction Layer
Data Grid
Clients
Data download
Query Abstraction
Browse / Display
Spatial Information
Textual Information
Data Discovery
GRASP Architecture
GRASP uses a data grid as an abstract storage repository.
Metadata in the grid is mined from the grid itself or from external sources and published into a browsable form.Data grids may allow for platform independent
metadata, but may not be optimal for access
Grid Holdings
Registered GLCF holdingsOver 338,000 registered files4.4Tb total sizeGranular permissions on registered holdings
No need to move all data into grid, registered pre-existing holdings in place.
Current data grid
Designed using newer SRB software that allows for Federated grid model.
Created using standard SRB software configuration (postgreSQL)
Large data sites can maintain independent MCATs Administratively independent Ability to customize data grid to site security/data
requirements while maintaining compatibility across federation.
Smaller peers can register as clients of UMD
Data grid overview
UMD MCAT
SRB Master
DFS Gateway
SRB Master
Local Peer
Remote MCAT
SRB Master SRB Master
SRB Master
Zone: esip-umiacsZone: esip-remote
GRASP Interface
Growing data grid
Additional MCATs can be federated Additional SRB aware clients can be
added Remote data supplying sites can
contribute data sets from their resources Sites needing quick local access to data
can replicate to local SRB setup.