A PPARC funded project AstroGrid and OGSA-DAI: Early experiences Kona Andrews Guy Rixon Institute of...

A PPARC funded project

AstroGrid and OGSA-DAI: Early experiences

Kona AndrewsGuy Rixon

Institute of Astronomy University of Cambridge

OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004

Data Set Access: The DataCenter

DataCenter component

DataAccessPlug-in

Data

Web Services

Tomcat / Axis


DataCenter JDBC plug-in


JDBC

DBMS

Web Services

Tomcat / Axis


DataCenter custom plug-in


File querier

FITS files

Web Services

Tomcat / Axis


DataCenter proxy plug-in


Service proxy

Vizier

Web Services

Tomcat / Axis


What we imagined

OGSA-DAI

Web Services

DataCenter Component

OGSA-DAI client plug-in

Tomcat / Axis

Grid Service Ports

Tomcat / OGSA


Problem

OGSA-DAI

Web Services


OGSA-DAI client plug-in

Tomcat / Axis

Grid Service Ports

Tomcat / OGSA


Nasty solution

OGSA-DAI

Web Services


OGSA-DAI “client” plug-in

Tomcat / Axis

Grid Ports

Tomcat / OGSA

OGSA-DAIclient

Separate JVM


Current use of OGSA-DAI activities

DataCenter

OGSA-DAI

PostgresDBMSADQL



DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQLADQL

SQL



DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQL

sqlQueryStatement

ADQL

SQL SQL



DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQL

sqlQueryStatement

ADQL

SQL

Result rows

SQL

(to XML RowSet)



DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQL

sqlQueryStatement

deliverFromURL

(to XML RowSet)

ADQL

SQL

Result rows

SQL



DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQL

sqlQueryStatement

xslTransform

deliverFromURL

(to XML RowSet)

ADQL

SQL

Result rows

SQL

VOTable



DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQL

sqlQueryStatement

xslTransform

deliverToFile / deliverToGFTP

deliverFromURL

(to XML RowSet)

ADQL

SQL

VOTable

Result rows

SQL



DataCenter

OGSA-DAI

PostgresDBMS

ADQL to

SQL

sqlQueryStatement

xslTransform

deliverToFile / deliverToGFTP

deliverFromURL

(to XML RowSet)

ADQL

SQL

VOTable

Result rows

SQL

Deliver results


Comments

Astronomers often need LARGE result sets

Thousands to millions of rows

Memory usage is a big issue

OGSA-DAI 3.1 much better than 3.0

xslTransform activity hogging memory


OGSA-DAI speed penalty

0

10

20

30

40

1e3 1e4 1e5 2e5 4e5 1e6 2e6

Number of result rows

Rough figures only

Measured without xslTransform activity

Will likely vary with available memory


Planned extension: User tables

OGSA-DAI

DBMSDataCenter

R/ODatasets

Query Query results



OGSA-DAI

DBMSDataCenter

R/ODatasets

Usertables

Query Query results Data flow

GDS to GDS transfer

Bulk load

OGSA-DAI


DBMS


OGSA-DAI

DBMSDataCenter

DataCenter

OGSA-DAI

Table Manager

R/ODatasets

Usertables



DBMS


OGSA-DAI

DBMSDataCenter

DataCenter

OGSA-DAI

Table Manager

Client

User data

R/ODatasets

Usertables



Comments

Will be relying on sqlBulkLoadRowSet (!)

Table management facilities currently thin

Not using X.509 auth at user granularity

Still require W/S-based access to G/S


Our needs

Ability to handle very large result sets - Don’t rely on loading whole result set into memory!

Suite of DB/table management activities- List tables in DB

- Describe table column types / space usage

- Last access/modification time

- Etc etc etc

Ability to create and delete indices on tables

Ultra-simple installation procedures


The End

www.astrogrid.org

A PPARC funded project AstroGrid and OGSA-DAI: Early experiences Kona Andrews Guy Rixon Institute of...

Documents

Transcript of A PPARC funded project AstroGrid and OGSA-DAI: Early experiences Kona Andrews Guy Rixon Institute of...