TASKFLOW: STATE

MANAGEMENT FRAMEWORK

Vishal Yadav

NEC Technologies India Ltd.

Agenda

• Motivation for StateManagementics

• Expectations

• Scenario-1: VM Boot Abort & Resumption

• Scenario-2: Transaction Example

• Here Comes the TaskFlow

• Foundation Concepts

• Patterns

• Engines

• Persistence

• TaskFlow Demo

• Reference

• Questions

2

Motivation for StateManagementics

• Workflows without state management in place makes these hard

to follow, alter and recover

• Unreliable workflow and resource state

It is hard to maintain correctness and consistency of distributed

systems

• RPC boundaries are a constant balance of improving scalability

but decreasing consistency

• Race conditions occur more often than desired

• Manager Driver API boundary

• Application and state recovery typically patched on after the fact

instead of built-in from the ground up (e.g. periodic tasks)

• Capability to service stop an application cleanly without manual

(or periodic) clean up is crucial for features like live upgrades

3

Expectations

• Customers expect stability and consistency

API and service reliability

• Resource and/or state corruption (or people to fix manually these

problems) costs $$$

• Easy understanding of workflows and states allows the

development and alteration of existing workflows

• Upgrades (not even live), just upgrades

Just say no to destroy the cloud to upgrade

4

5

Scenario-1: VM Boot Abort & Resumption

6

Scenario-1: VM Boot Abort & Resumption

We can continue the run instance process again from the point

where flow failed.

7

Scenario-2: Transaction Example

Flow resemblance with SQL transaction.

START TRANSACTION

task1: call API-1 to do op-1 || ROLLBACK

task2: task1 finish, call API-2 to do op-2 || ROLLBACK

...perform other tasks...

COMMIT

8

Here Comes the TaskFlow

• A library that makes task execution easy, consistent, and reliable

• Provides a framework for executing flow of operations such that

those flows can be paused, resumed, revoked at any time

• A paradigm and lightweight framework

• Community driven & well documented

• Library and code available though pypi & github

https://pypi.python.org/pypi/taskflow

https://github.com/openstack/taskflow

9

Foundation Concepts

• Code structure (your applications frame)

• Controlled execution

Who & what manages the overall execution

• Persistence (how you know what was executed)

• Work recovery

How you recover from failure/partial progress

10

Foundation Concepts

11

• Tasks

Executes and reverts one action

Receives inputs and declares outputs

• Flows

Composes tasks (or subflows) into useful structures

Imposes some definition of order also called patterns onto

the running of your tasks or subflows

Foundation Concepts

12

• Jobs

The initial (and any derivative) set of tasks & flows required to

fulfil an action

Can be transferred to a worker for completion

Can be re-associated on worker failure (or timeout) for

resumption or undo/reversion

• Job Board

A system where jobs can be atomically posted, reposted,

claimed, marked as completed

Backed by varying implementations

Message queue, zookeeper, database…

Foundation Concepts

13

Patterns

Attributes/ Patterns

Linear Unordered Graph

Summary Runs a list of tasks/flows, one after the other in a serial manner

Runs a set of tasks/flows, in any order

Runs a graph composed of tasks/flows in dependency driven ordering

Constraints Predecessor tasks outputs must satisfy successive tasks inputs

Disallows inter-task dependencies

Dependency driven, no cycles. A tasks dependents are guaranteed to be satisfied before the task will run

Use-Cases This pattern is useful for structuring tasks/flows that are fairly simple, where a task/flow follows a previous one

This pattern is useful for tasks/flow that are fairly simple and are typically embarrassingly parallel

This pattern allows for a very high level of potential concurrency and is useful for tasks which can be arranged in a directed acyclic graph (without cycles).

14

Attributes/ Patterns

Linear Unordered Graph

Benefits Simple Simple. Inherently concurrent

Allows for complex task ordering. Can be automatically made concurrent by running disjoint tasks/flows in parallel

Drawbacks Serial, no potential for concurrency

No dependencies allowed. Ease of tracking and debugging harder due to lack of reliable ordering

Complex. Can not support cycles. Tracking and debugging harder due to graph dependency traversal

Patterns

15

• Runs flow (and associated tasks) in a well defined, reliable,

consistent and resumable manner

Follows well defined state transitions

• Allows for deployers/developers of a service that uses taskflow to

select an engine that suites their setup best

• Backed by varying implementations

Single-threaded

Multi-threaded via native or green threads

Distributed

• Support for other engines:

Message based(RPC), oslo.messaging

Celery

Engines

16

• Saves task state/progress/results and flow state

• Allows for reconstruction and resumption of flows and associated

tasks

• Allows the user to view the play-by-play action history of flows

and associated tasks

Facilitates debugging of taskflow usage and integration

• Backed by varying implementations

File system, memory, database…

Persistence

17

TaskFlow Demo

• Pre-requisite: Download taskflow library.

• Example: taskflow_linear_example.py

https://gist.github.com/chadlung/eea627fab8b26a869f4d#file-

fetch_example_linear-py

$ pip install taskflow

18

Questions??

19

Reference

• TaskFlow Documentation:

https://wiki.openstack.org/wiki/TaskFlow

docs.openstack.org/developer/taskflow

• TaskFlow Demo Example:

https://gist.github.com/chadlung/eea627fab8b26a869f4d#file-

fetch_example_linear-py

20

21