ZigZag: The Meandring Language

ZigZag: The Meandring Language

Xavier Llorà"

Data-Intensive Technologies and Applications,!National Center for Supercomputing Applications, !

University of Illinois at Urbana-Champaign

[email protected]

The SEASR project and its Meandre infrastructure!are sponsored by The Andrew W. Mellon Foundation

Powered by 2 Xavier Llorà

Outline

•  Motivation

•  Do we need yet-another language?

•  ZigZag: A language for describing DG data-intensive flows

•  Syntax

•  ZigZag @ Work

Motivation

•  Meandre success relies on the ability to easily assemble data-intensive flows

•  Meandre Workbench is:

–  A visual tool for assembling data-intensive flows

–  A tool useful for non-technical users

–  Oriented to interactive usage

–  Does not allow easy orchestration


Motivation

•  Speed up the flow development cycle:

–  Data-intensive flows are mainly assembled using the WB

–  Takes a long time from you start building the flow till you run it

–  Does not allow quick repetitive flow tuning and large-scale experimentation

•  Large-scale job orchestration:

–  Self contained units make portability easier

–  It is not feasible using the WB approach

–  Before ZigZag, Meandre did not provide any scripting language


Why Do We Need Yet-Another Language?

•  Provide a fast-lane for data-intensive flow development:

–  Describe a data-intensive flow (directed graph)

–  Compile a data-intensive flow into a self-contained task

–  Run the task as a regular job

•  Large-scale tasks require basic movable units

•  Streamline the tools and the process for hard-core developers

•  Prepare the stage for distributed data-intensive flow execution


ZigZag: A language for Describing "DG Data-Intensive Flows

•  A simple language for describing data-intensive flows

•  Modeled after Python’s simplicity

•  ZigZag is a declarative language for expressing the directed graphs

•  ZigZag also provides a compiler to transform a ZigZag script (.zz) into a self-contained task

•  A Meandre self-contained task—Meandre archieval unit (.mau)—can be executed by the Meandre engine

•  Command-line tools include: (1) a console for interactive scripting, (2) a ZigZag compiler, and (3) a ZigZag run environment


ZigZag Streamlined Process


.zz ZigZag Compiler .mau 1

.mau ZigZag Run Environment

STDOUT 2

.zz .zz .zz .zz

.mau .mau .mau .mau

ZigZag Compiler

ZigZag Run Environment

STDOUT STDOUT STDOUT STDOUT

ZigZag Console

3

ZigZag Syntax

•  Pythonic case-sensitive language

•  Four basic instructions

1.  Component discovering and aliasing (CDA)

2.  Component instantiation (CI)

3.  Instance modification (IM)

4.  Instance invocation (II)

•  Instructions are grouped in .zz files

•  Each .zz file may describe one data-intensive flows

•  Comments start at the beginning of a line using the # symbol


ZigZag: Component Discovery and Aliasing

•  CDA instructions

–  Retrieve components from a repository

–  Allow creating convenient aliases names for components

•  The basic retrieval presents two forms

–  import <URL_to_the_repository>

–  from <URL_to_the_repository> import <component_URI>

•  CDA alias instructions also present two basic forms

–  alias <component_URI> as COMPONENT_ALIAS_NAME

–  from <URL_to_the_repository> import <component_URI> as COMPONENT_ALIAS_NAME


ZigZag: Component Discovery and Aliasing

•  CDA examples:

import <http://demo.seasr.org:1714/public/services/demo_repository.ttl>

alias <meandre://test.org/component/push-string> as PUSH

alias <meandre://test.org/component/to-uppercase> as TOUPPER

alias <meandre://test.org/component/print-object> as PRINT

from <http://demo.seasr.org:1714/public/services/demo_repository.ttl>

import <meandre://test.org/component/push-string> as PUSH

from <http://demo.seasr.org:1714/public/services/demo_repository.ttl>import <meandre://test.org/component/to-uppercase> as TOUPPER

from <http://demo.seasr.org:1714/public/services/demo_repository.ttl>import <meandre://test.org/component/print-object> as PRINT


ZigZag: Component Instantiation

•  CI instructions create component instances"

–  instance_name = COMPONENT_ALIAS_NAME() "

•  Supports multiple assignments"

–  ins1, ins2 = COMPONENT_ALIAS_NAME(), COMPONENT_ALIAS_NAME()


ZigZag: Component Instantiation

•  CI examples:

push_hello = PUSH()

toupper = TOUPPER()

print = PRINT()

push_hello, toupper, print = PUSH(), TOUPPER(), PRINT()


ZigZag: Instance Modification

•  IM instructions allow to modify property values of a components instance

–  instance_name.property_name = “value”

–  instance_name.property_name = another_instance.property_name

•  Supports multiple assignments too

–  instance_name.property_name,instance_name.property_name = “value”, another_instance.property_name


ZigZag: Instance Modification

•  IM examples:

push_hello.message = “Hello World!!!”

push_hello.times = “10”

push_hello.message = push_another_hello.message

push_hello.message, push_hello.times = “Hello World!!!”, “10”


ZigZag: Instance Invocation

•  II instructions

–  Describe instance execution invocation

–  Define the DG graph of the data-intensive flow

–  Use the object invocation abstraction to define data port interconnection

–  The semantic parsing guarantees the constrains of data-intensive flows

•  Allows single and multiple assignments

•  The left-hand side of the assignment is a pseudo-object identifying output ports

•  The right-hand side invokes and links ports of instances



•  Dictionary/Keyword-based invocations allow implicit definition of the DG structure of the data-intensive flow"

–  @pseudo_object = instance_name ( input_port_name_1: pseudo_object.output_port_name_1; … input_port_name_n: pseudo_object.output_port_name_n )

•  Multiple assignments are also allowed



•  II examples:

@hello = push_hello()

@upper = to_upper(string:hello.string)

print(object:upper.string)


ZigZag: All Together

import <http://demo.seasr.org:1714/public/services/demo_repository.ttl>

alias <meandre://test.org/component/push-string> as PUSH

alias <meandre://test.org/component/to-uppercase> as TOUPPER

alias <meandre://test.org/component/print-object> as PRINT

push_hello, to_upper, print = PUSH(), TOUPPER(), PRINT()

push_hello.message, push_hello.times = "Hello World!!!", "10"

@hello = push_hello()

@upper = to_upper(string:hello.string)

print(object:upper.string)


ZigZag @ Work

•  Copy the previous example to a file called push_hello.zz

•  Get a copy of the ZigZag console and place them together

•  You can download the latest tools here

–  http://seasr.org/meandre/download/


ZigZag @ Work


ZigZag @ Work

•  Start the ZigZag console

–  $ java -jar zz-1.4.0-vcli.jar


ZigZag @ Work


ZigZag @ Work

•  Load the ZigZag file (you will need to have access to the internet to load the components from the repository)

–  >>> load zigzag push_hello.zz

•  You should now be able to list the new flow instances by typing

–  >>> ls

•  To get help about the console commands, just type

–  >>> help


ZigZag @ Work


ZigZag @ Work

•  The console provides an interactive visualization of the flow being constructed so far

•  To visualize the flow type

–  >>> show flow


ZigZag @ Work


ZigZag @ Work

•  You can now save a .mau file containing the assembled flow by typing

–  >>> save mau push_hello.mau


ZigZag @ Work


ZigZag @ Work

•  You can also run the created flow by just typing

–  >>> run


ZigZag @ Work


ZigZag @ Work

•  You can also compile the ZigZag file using the command-line tool

–  $ java -jar zzc-1.4.0-vcli.jar push_hello.zz


ZigZag @ Work


ZigZag @ Work

•  The compiled ZigZag can be run using the command-line tool

–  $ java -jar zzre-1.4.0-vcli.jar push_hello.mau


ZigZag @ Work


ZigZag: The Meandring Language

Xavier Llorà"

Data-Intensive Technologies and Applications,!National Center for Supercomputing Applications, !

University of Illinois at Urbana-Champaign

[email protected]

The SEASR project and its Meandre infrastructure!are sponsored by The Andrew W. Mellon Foundation

ZigZag: The Meandring Language

Education

Transcript of ZigZag: The Meandring Language