Workflow: Getting Startedwith SMS & NED Developer Training

NASA SIVO Code 610.3

With GMAO

May 2008

Outline

• Review of Workflow System Concepts

• Example Workflow

• GEOS GCM Workflow

• Hands-on Example

• Troubleshooting

• Resources

Basic Workflow Terms

SMS - the workflow engine that runs the scriptsCDP - command line shell for SMSTrigger - an event that kicks off a taskDependency - usually synonymous with trigger

SMS Components• Suite - an instance of an SMS workflow

• Described by a .def file• Also is the top-level directory

• Family - a group of related Workflow tasks or other families• Described within a .def file• Represented as a directory

• Task - a unit of work in the Workflow• Described within a .def file• Defined as a .sms file

Files Making up a Workflow

• Workflow definition - SMS file thatdefines overall structure of workflow tasks& their dependencies

• Workflow scripts - SMS files thatrepresent tasks within a workflow; Oursgenerally call shell scripts

• Run scripts - any shell scripts thatperform an action for a workflow task

• Configuration files (.ned) - any sort offile generated from the variables definedin NED

Submitting a workflow to runin NED1. Local install directory created

• Fresh install of a workflow will go here

2. Workflow & run scripts are created• Currently under source control• Checked out from source

repository, has tags & branches• Added to the install directory

3. Configuration files arecreated

• Defines all the variables shownin NED for user to configure

• Generated according to theconfiguration settings

• Added to the install directory

4. Calls an initial “submission” shellscript

• Script, generally, will submit theworkflow to SMS

Workflow Example

Configuration

Running workflow

Workflow Example

Configuration in NED

Configuration in XML

Workflow ExampleWorkflow script - SMS file(runModelSegment.sms)

Run script - shell script(runModelSegment.bash)

Definition file(workflow.def)

Some SMS commands

GEOS Workflow

gcm_setup script(Larry’s script)

991 lines

~20 GEOS5 run scripts (coupledModel/GEOSgcm/bin)

Decompose into

setup_wf_env.bash

GEOS Workflow

• Access the GEOS5 workflow scripts from mapmeby typing at the prompt: >> getgeos

• coupledModel/GEOSgcm/bin contains the coreGEOS5 run scripts:

archive_restarts.bashcreate_exp_dirs.bashcreate_live_rcs.bashcreate_rc_templates.bashgetgcm_rst_exe.bashget_restarts.bashgsi_analysis.bashlinkbcs.bashload_batch_parms.bashmove_history.bash

post_processing.bashpull_back_data.bashregress_test.bashremove_wf_env.bashrename_chk_to_rst.bashrename_rst_for_ana.bashresubmit.bashrun_model.bashsetup_model_env.bashsetup_wf_env.bash

Hands-On Demo

Simple Demo to Modify

• Check out a demo workflow

• Make a simple modification

• Check it back in

• Run it

Requirements for RunningWorkflows

• Required for this demo:• Some experience with CVS or Subversion

• Account on mapme

• Member of wf_dev group on mapme

• Account on discover and/or explore

• A dirac account is not required for this demo

• User’s must have a dirac account to run theGEOS5 workflow

Setup a New Workflow

• Log-in to mapme• First time developers must switch to the “wf_dev”

group:newgrp wf_dev

• Create a working directory and cd to it.• Run developer startup script to add command-line

shortcuts:. /home/workflow/bin/script/wfdev.bash

• Execute the following scripts to create a new “demo”workflow in your working directory

branchWorkflow <username> checkoutWorkflow <username>

Enhance the Workflow

• Goal here is to add a new variable to theworkflow• cd to working directory/<username>/bin and edit

script runModelSegment.bash to add the variable“expName”.• Enter ‘echo “Variable: $expName”’.

• Check in the Demo workflow checkinWorkflow

Enhance the Workflow (cont.)

• Bring up NED• Select File -> Open from File. Click OK.• Navigate to working directory/<username>

• open baseWorkflow.ned .• Go to File -> Client Mode and select “Design Mode”• Under Configuration Tree select Variable Group. Execute

<ctl>right click and navigate to Add a Variable.• Enter expName in the information field of the Name property (hit

return to enter the string in NED)• Expand the Variable Group and note the variable named

expName listed with the existing variables.• Under Configuration Tree select Base Workflow.

• Execute <ctl>right click then navigate to Edit Workflow Properties.• In the Repository Branch workflow property enter:/baseWorkflow/branches/<username>

Enhance the Workflow (cont.)

• Save and run NED workflow• Select File -> Save to File. Choose a name for your new

NED workflow.• Select Run -> Submit Workflow

• Navigate to the tab named <Unique ID> and watch theworkflow execute.

Tips: Testing Workflows

• Write the workflow for easier testing• Be generous with echoes/prints to track problems faster

• Check for error codes in scripts

• Run the workflow to test it• New workflows get installed and run under:

/home/workflow/expWorkDir

• Check the install directory when problems occur

• Test the workflow• Incrementally test to fix problems as you develop

• Use multiple user accounts to avoid hard-wired solutions

Tips: TroubleshootingWhen a workflow fails…

• Determine where the problem is coming from• Files may be missing - did you commit?• SMS scripting errors - look at SMS job files• User script errors

• Check the log file• Edit and re-run the task to see if it passes

• Remote script errors - must have a log file

• Visit Modeling Guru for questions and answers• NASA website:

http://modelingguru.nasa.gov/clearspace/community/mapmewkflow

• SIVO staff regularly monitors Modeling Guru• Knowledge base for Earth System modeling

• Contact SIVO workflow staff for more assistance

Other Resources

• Modeling Guru communityThis points to a number of resources for workflow developers, such

as discussion forum that you can use to post questions to us andothers.User tutorialsWorkflow group documentation

https://modelingguru.nasa.gov/clearspace/community/mapmewkflow

• SMS DocumentationThe SMS software was built by ECMWF, so while it is helpful, it may

refer to things specific to their environment, like MARS orprepIFS.

http://www.ecmwf.int/publications/manuals/sms/documentation/index.html

BACKUP SLIDES

Steps to integrating the GEOSrun scripts

1. GEOS run.script is componentized or decomposed• Identify portions of the script that are generally

• Independent,• Perform a basic task

• Pull out that portion into a new script

2. Associate with SMS• New script becomes an SMS task• Task is defined in the definition file• A .sms file is created that invokes the shell script

3. Finally add variables from the workflowconfiguration

• Most scripts will require user-configurable variables• Simply source the file generated by NED at the top of the

script

Special cases

• Scripts that finish running are considered“complete” by SMS• So submitting a job to a queue requires you to

wait for the job to complete before exiting

• Scripts that get extracted may have variabledependencies• Shared variables between scripts can be included

• Breaking down tasks changes how jobs enterthe queue• Lots of small jobs vs. single large job

Workflow Config Management

Ideally GEOS source and workflow’s run scriptsshould be kept in sync in the same repository

Workflow System in NCCS

• Run from singlelocation withinNCCS: “mapme”

• Connects to HPCresources toperform tasks• palm• discover• dirac• Others

Workflow System in NCCS