Slide 1Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

AstroGrid’s trial Data Grids

Experiences with GT3

Guy Rixon

July 2003

Slide 2Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

Themes

• Grid metaphor; approaches

• OGSI cone-search services

• OGSA-DAI

• OGSI image-servers

• “MySpace”

• The quality problem

Slide 3Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

What does “data grid” metaphor mean?

• Complex

• Heterogenous

• Hidden

• Simple

• Uniform

• Accessible

Or

Slide 4Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

What’s the commodity?

• Compute cycles:

• S/w installations:

• Archived data:

• Views of data:

• Storage:

• Transport:

“Grid” implies and requires some commodity!

Slide 5Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

Types of service

On-line services

SOAP OGSI OGSA-DAI

Data-selectionservices

OGSA-DAI

Trad. Data centres

A/G 2002 prototypes

A/G 2003 prototypes

Slide 6Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

Cone search services• Data-selection grid services

– For tabular data

– Initially three data-sets

– One service per data set

• Metadata only to browser

• Bulk results via file servers– Only HTTP.

– No data consumers!

Slide 7Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

Cone-search services (2)• Semi-OGSI construction:

– Built with Globus Toolkit 3 alpha 2.– Java grid-services in Tomcat call Perl scripts.– RPC-style operations.

• Data management built into services– Done by local scripts, not by grid services– Conflates data and service grids

• Easy coding!– Services: trivial– Clients: harder

Slide 8Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

OGSA-DAI

• Brief flirtation, ran out of time.• Successfully wrapped FIRST object-catalogue

(~80MB, 750, 000 rows)• Tried to make cone-search client: too hard!• Data grid:

– Not possible with OGSA-DAI 1.5

– Try again with v2.x

• Document-based service: hard to write clients for.

Slide 9Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

Anglo-Australian demo• Drives a volumetric renderer with service and data grids.

• In collaboration with AusVO (Barnes et al.)

• To be shown at IAU 2003.

Slide 10Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

Anglo Australian Demo (2)• Uses Globus Toolkit 3 beta.• Uses GridFTP.• RPC services throughout.• Stateful (tidies up locks and temp. files).• Some security.• Infrastructure grief limits development.• See it at:

– IAU 2003– Super Computing 2003– All-hands 2003

Slide 11Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

“MySpace”

Slide 12Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

“MySpace” (2)

• Pure data-grid.• Handles files and relational tables.• Tree of logical files spans VO.• Permanent “home” space + leased “scratch” space.• Relational (OGSA-DAI) nodes embedded in tree.• One MySpace agent at each service site.• Central registry of MySpace contents.• MySpace Explorer; c.f Windows Explorer.

Slide 13Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

Lessons

• Services are easy; clients are hard.• Registry matters, even at trivial scale.• Web portal is very hard.• Quality threshold is very high.

– Small errors get magnified in big grid.– State holding makes it hard to recover errors.– System is only ever as good as its error reports.– Infrastructure failures hard to trace and fix.– Results limited by infrastructure quality.– GT3: arghhhhhhhhhh! Noooooo!

Slide 14Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

The quality crisis

Old style…

You are here

New style…

You are here(old tech.)

You are here(new tech.)

Stateful, distributed, semi-autonomous systems demand extreme quality.Current Grid infrastructure does not meet this standard.

Slide 15Guy Rixon: AstroGrid’s trial data-grids “VO as data grid”: NeSC, July 2003

Responses to quality crisis• Wait for better infrastructure?

– But: “Good software takes 10 years” — Spolsky

• Avoid grid? Or just GT3?• Give up?• Simplify?• Take control:

– Make own infrastructure to sufficient standard– Get APIs standardized, then done properly.– (Delegate to industrial partners?)– Only return to application programming when infrastructure is

sound.