Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA [email protected].

37
Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA Allan Doyle, NASA adoyle@intl- adoyle@intl- interfaces.com interfaces.com

Transcript of Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA [email protected].

Page 1: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Workflow + Globus Toolkit Update

September 8, 2004

Allan Doyle, NASAAllan Doyle, NASA

[email protected]@intl-interfaces.com

Page 2: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

2

AgendaAgenda

• Workflow activity• Globus Toolkit status• CA Activity (Yonsook)

Page 3: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

3

AgendaAgenda

• Workflow activity• Globus Toolkit status• CA Activity (Yonsook)

Page 4: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

4

Workflow ActivityWorkflow Activity

• Began in late 2003• Goal was to be able to chain

processing among participants in a single workflow

• Held a series of teleconferences• http://grid-tech.ceos.org/gridwiki/WorkflowTige

rTeam

(Ask for username/password)

Page 5: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

5

Workflow activity - 1Workflow activity - 1

• First, we learned about workflow and about CEOS Grid Participants’ workflow tools…

Page 6: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

6

Workflow overviewWorkflow overview

• The next slides are taken from Tech Team members’ presentations.

• Annotations that were added for this presentation look like this

Annotation box…

Page 7: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

7

What is a Workflow?What is a Workflow?

Dutch Space has renamed their

workflow tools to GridAssist…

Page 8: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

8

Workflow LevelsWorkflow Levels

Application Domain

Abstract Workflow

Concrete Workflow

Execution Environment

Application Component Selection

Resource SelectionData Replica Selection

Transformation Instance Selection

= Data Transfer

FFT

FFT filea

host2

host1Retry

Pick DifferentResources

Specify a DifferentWorkflow

copy host1://home/filea to host2://u1/file1

/usr/local/bin/fft /u1/file1

From Pegasus presentation by Ewa Deelman, ISI

This material was redrawn to enhance visibility, it’s

from Ewa Deelman’s workflow presentation…

Page 9: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

9

Typical Workflow EnvironmentTypical Workflow Environment

Page 10: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

10

Workflow exampleWorkflow example

This is a domain (or application level)

workflow…

Page 11: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

11

Workflow detailed exampleWorkflow detailed example

This is an abstract workflow…

Page 12: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

12

GMU Virtual Product ConceptGMU Virtual Product Concept

Virtual data concept requires workflow tools in

order to be built…

Page 13: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

13

Workflow progressionWorkflow progression

Different levels of workflow…

Domain… Abstract… Concrete…

Page 14: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

14

Workflow toolsWorkflow tools

• DutchSpace - Grid Assist– Developed in-house, workflow engine and

client tools. (Runs on Globus Toolkit 2.x)

• NCSA (UAH partner) - Open GCE Runtime Engine– Based on Apache Ant (not Globus dependent)

• ISI - Pegasus– Developed as part of Earth Systems Grid work

w/Argonne (Runs on Globus 2.x, 3.x?)

• GMU - GeoTree– Based on BPEL (Business Process Execution

Language) (Runs on Globus 3.2.x)

Page 15: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

15

Workflow Activity - 2Workflow Activity - 2

• Then we investigated workflow interoperability.

• The plan was to1. Ask participants for use cases2. Decompose use cases into workflow

patterns

Page 16: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

16

Use case descriptionsUse case descriptions

• Identifier• Description• Actors• Goal• Preconditions• Assumptions• Frequency

• Basic Course• Alternate Courses• Included Use

Cases• Extended Use

Cases• Issues• Decisions

Workflows can be derived from these descriptions of actions

Page 17: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

17

Workflow PatternsWorkflow Patterns

• http://tmitwww.tm.tue.nl/research/patterns/

• Provides a good explanation and definition of 21 workflow patterns.

• Next slide shows some of the simple ones as examples.

Page 18: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

18

Basic workflow patternsBasic workflow patterns

A B CSequence: B is started after A completes. C is started after B completes.

A ANDB

C

Parallel Split: B and C are begun any time after A has completed.

CANDA

B

Synchronization: C is not begun until A and B have both completed.

A XORB

C

Exclusive Choice: Following A, a choice is made to pursue B or C but not both.

CXORA

B

Simple Merge: C activated once for every completion of A or B

Page 19: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

19

Use case analysisUse case analysis

• Use cases were received from– USGS– UAH/NOMADS– Dutch Space

• USGS and UAH/NOMADS use cases were refined via teleconference + email

• All are available on Workflow web page

Page 20: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

20

Initial FindingsInitial Findings

• Workflow tools are a necessary component!– Grid applications are partioned into work units

that run on many machines.– Runtime error handling is a vital part of

workflow engine capability

• Grid workflows + tools are complex– Tools take a long time to learn– Grid workflows support complex processing

tasks

• Many (~30) different Grid workflow systems exist– Interoperability among workflow tools is

currently nonexistent

Page 21: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

21

Use case findings - 1Based on the USGS and NOMADS/UAH Use Cases

Use case findings - 1Based on the USGS and NOMADS/UAH Use Cases

• Found that use cases tended to have many “hands-on” steps where scientists would make choices– This is one of the key areas the Grid workflow

community has identified as needing work.

• Use of “formal” workflow tools may be more work than it’s worth for the use cases– Application projects are using the Grid mainly

for data transfer, not computing– Workflows include some automation areas

(reprojection, reformatting, subsetting).

Page 22: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

22

Use case findings - 2Based on the Dutch Space Application description

Use case findings - 2Based on the Dutch Space Application description

• Dutch Space application is well suited for formal tool

• Dutch Space has implemented their own tool in response to their need and the (then) lack of existing tools.

• OMI application has long processing chain (c.f. earlier slides)

Page 23: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

23

Workflow ConclusionsWorkflow Conclusions

• Workflow community is starting to talk about interoperable workflows– http://www.extreme.indiana.edu/groc/

ggf10-ww/index.html

• Perhaps cross-workflow interoperability is not an immediate need– A single workflow tool can be used to control

remote job execution without needing to interoperate with other workflow tools.

– CEOS Grid activity should wait until Application Projects have a clear need for cross-tool workflow interoperability before doing further work.

Page 24: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

24

AgendaAgenda

• Workflow activity• Globus Toolkit status• CA Activity (Yonsook)

Page 25: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

25

Globus Toolkit (GT) UpdateGlobus Toolkit (GT) Update

• Current GT version is 3.2– Most CEOS Globus users are still using 2.4

• GT version 4.0 release is scheduled for January 31, 2005– http://www-unix.globus.org/toolkit/GT4Facts/

• GT4 will be backwards compatible with GT3.2

• Next 5 slides taken from the presentation given in Tromso, also available at

http://grid-tech.ceos.org/gridwiki/GlobusVersions

Page 26: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

26

Page 27: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

27

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Page 28: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

28

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Page 29: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

29

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Page 30: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

30

Page 31: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

31

GT 4.0 schedule and contentGT 4.0 schedule and content

• Next two slides have material from http://www-unix.globus.org/toolkit/GT4Facts/

Page 32: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

32

GT4 Release ScheduleGT4 Release Schedule

Page 33: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

33

GT4 ComponentsGT4 Components

Page 34: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

34

Globus Toolkit RecommendationsGlobus Toolkit Recommendations

• If you are using– GT 2.4

• Migrate to 4.0• Don’t go to 3.2 as intermediate step

– GT 3.2• Migrate to 4.0• 4.0 will be backwards compatible

Page 35: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

35

AgendaAgenda

• Workflow activity• Globus Toolkit status• CA Activity (Yonsook)

Page 36: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

36

CA ActivityCA Activity

• Following earlier work on Certificate Authorities (CA), NASA IPG (Information Power Grid) volunteered to provide CA services for CEOS Grid.

• NASA IPG funding has been cut– CEOS Grid cannot rely on IPG CA

anymore

• We decided to establish a CEOS CA on a NASA machine at GSFC

Page 37: Workflow + Globus Toolkit Update September 8, 2004 Allan Doyle, NASA adoyle@intl-interfaces.com.

Sept 8, 2004Workflow + Globus Update

37

CA Activity - cont.CA Activity - cont.

• Ananth Rao has installed a test CA and has produced a Certificat Request tool similar to the one provided by IPG

• This tool will undergo further testing after WGISS-18

• Once we are satisfied with it, we will provide it to the CEOS Grid