Give your little scripts big wings: Using cron in the cloud with Amazon Simple Workflow

30
Asad Jawahar Give your little scripts big wings: Using cron in the cloud with Amazon Simple Workflow Senior Technical Program Manager

description

Most developers write them and every company has them – a vast library of small and large scripts that are designed to run on a scheduled basis. These background angels help keep the lights on and the doors open. They’ve been built up over time and are forgotten little heroes that are only remembered when the machines they live on fail. They are scattered throughout a company’s IT infrastructure and do important things. In this session, we will explain how to use Ruby on Simple Workflow to quickly build a system that schedules scripts, runs them on time, retries them if they fail, and stores the history of their execution. You will walk away from this session with an understanding of how Simple Workflow brings resiliency, concurrency, and tracking to your applications.

Transcript of Give your little scripts big wings: Using cron in the cloud with Amazon Simple Workflow

Page 1: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Asad Jawahar

Give your little scripts big wings: Using cron in the cloud with

Amazon Simple Workflow

Senior Technical Program Manager

Page 2: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

For applications with multiple connected steps…

• Amazon Simple Workflow provides the building blocks and a controller to

reduce the complexity of infrastructure programming and state machinery.

This helps customers focus on...

• Algorithms and logic that make their organizations unique

Page 3: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Many Use Cases

Page 4: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Cron

• Scheduled tasks

• Why use SWF for Cron?

– Failure handling

– Scale

– Lost tasks (office move, machine failure)

– OS provided cron not an option (shared hosting)

Page 5: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Roadmap

c c : code The implementation of

your control logic and

steps

d d : deployment worker processes,

machines, SWF

processing engine

l l : logical process model, flow

control, retry logic

etc

→ →

Page 6: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Key Simple Workflow concepts

Activities (and workers)

Workflows (and workers)

• Discrete steps in your application, processors

• Logical grouping for multiple activities, processors

Deciders • Control logic for workflows:

execution order, retry policies, timer logic, etc. – Decision tasks

Retained workflows • Execution history for

workflows: auditable, re-playable

Timers, signals • Scheduled actions, Workflow

“interrupt requests”

Page 7: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

SWF Concepts

Workflow – Image processing Decider logic Activities

Download

Encode

Apply DRM Upload

Copyright?

Amazon SWF

Activity Workers

Workflow Workers

Decision Tasks

Activity Tasks

Activity Tasks

Activity Tasks

History

Page 8: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Responsibilities

AWS You

SWF

• Step sequence logic (“secret sauce”)

• Discrete step logic • Workflow workers • Activity workers • I/O data

• State of execution • Pending tasks • Execution history • Searching histories • Control engine

Page 9: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

class CronWorkflow

extend Decider

entry_point :start

activity_client :activityClient do |options|

options.prefix_name = "CronActivities"

end

def start(arg)

while true do

create_timer(86400){activityClient.backup “myDB"}

end

end

end

Cron in SWF

class CronActivities

extend Activity

activity :backup

def backup(arg) # backup script end

end

Page 10: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Daily Build Process Ordinary cron, single point of failure

Download files from S3

Run build tools

Upload build artifacts to

S3

Delete local files

Send email

Page 11: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Handling Failures

Download files from S3

Run build tools

Upload build artifacts to

S3

Delete local files

Send email

Pass?

Pass?

Yes

Yes No

No

Page 12: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Resilient Cron

Job1

Job2

Job3

Job4

Page 13: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

SWF

Resilient Cron

Distributed asynchronous interactions

Failure handling

Scalability Latency Auditability

• Coordinate scheduled tasks across distributed hosts

• Reliable messaging

Coordination engine in the cloud

• Stores and dispatches tasks • Delivery guarantees

• Add workers at any time • Need stateless workers • State provided by another system

Scalability

• Repository of distributed state

• Workers poll for work • Exactly once delivery

• Many types of failures • Different mitigations • Must react quickly

Fault Tolerance

• No loss of distributed state • Supports timeouts on work • Explicit failure modeling

• Need visibility into process execution

• Retain records for investigation

Audit History

• Index of executions • Play-by-play history of

execution • Retention of completed

executions

• Get work quickly • Route tasks to specific workers

Low Latency

• Long polling for work • Task routing through task

lists

Requirements

SWF provides

Page 14: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Introducing AWS Flow Framework (Ruby)

• Ease of use library for Amazon SWF APIs

• Uses standard Ruby constructs

• Open source

• Packaged as a gem

• In Private Beta (stay tuned for release)

Amazon

SWF

Page 15: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Benefits of AWS Flow Framework

• Run and load balance work on many machines with minimal changes

• Simplifies remote, long running, non-blocking steps

• Uses Amazon SWF to:

– Keep track of progress

– Triggers your code to execute next steps

• Failure handling built in

• Easily evolve your logic without adding complexity

Amazon

SWF

Page 16: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Hourly Build – Logical Control Flow

wait (1 hour)

copy files

run build task

upload files

delete local files

send email

if (failed)

retry up to 3 times

repeat

Download files from S3

Run build tools

Upload build artifacts to

S3

Delete local files

Send email

Page 17: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Build Cron Workflow – Execution Flow

Amazon SWF

Execution History

- Input data

- Download complete

- Build complete

- Upload complete

Decisions:

1. Schedule download [shared]

2. Schedule build [worker1]

3. Schedule upload [worker1]

DECIDER Makes decisions on what tasks to schedule, when, in what order

Start Workflow Execution

Your App, SWF Console or CLI

Starts execution of Cron Workflow

Worker

Worker

Long poll

Long Poll

Long Poll

Worker 2

Worker 1

Decision Tasks

Get task

Get task

1. /tmp, worker1

Return decisions

Shared

- Delete local file

- Email sent

2. Built

4. Schedule delete files [worker1]

5. Schedule email [worker2]

6. Execution complete

3. Uploaded 4. Deleted

5. Email sent

Get task

Page 18: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Hourly Build – Decider

class BuildWorkflow

extend Decider

entry_point :start

activity_client :client do |options|

options.prefix_name = "BuildActivities"

end

def start(source_bucket, target_bucket)

while true do

create_timer(25) { start_build(source_bucket, target_bucket)} end

end

def start_build(source, target)

begin dir = client.download(source) client.build(dir) client.upload(dir, target) ensure client.delete(dir) client.send_email(dir) end end

Page 19: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Task Routing

def start_build(source_bucket, target_bucket) activity_client :client do |options| options.prefix_name = "BuildActivities" end host_specific_task_list, dir = client.download bucket client.build(dir) do |options| options.default_task_list =host_specific_task_list end client.upload(dir, target_bucket) do |options| options.default_task_list =host_specific_task_list end client.delete(dir) do |options| options.default_task_list =host_specific_task_list end end end

Page 20: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Exponential Retry

def start_build(source_bucket, target_bucket) activity_client :client do |options| options.prefix_name = "BuildActivities" end dir = client.exponential_retry (:download, bucket) do |options| options.maximum_attempts = 3 end client.build(dir) client.exponential_retry (:upload, dir, target_bucket) do |options| options.maximum_attempts = 3 end client.exponential_retry (:delete, dir) do |options| options.maximum_attempts = 3 end end

Page 21: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Activities

class BuildActivities extend Activity

activity :download, :build, :upload, :delete do |options|

options.default_task_list = "list1"

options.version = "1"

options.default_task_heartbeat_timeout = "3600"

options.default_task_schedule_to_close_timeout = "30"

options.default_task_schedule_to_start_timeout = "30"

options.default_task_start_to_close_timeout = "30"

end

def download(bucket)

puts bucket

end

def build(dir)

puts dir

end

def upload(dir, bucket)

puts bucket

end

def delete(dir)

puts dir

end

end

Page 22: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Multiple builds in parallel

• Parent Cron workflow kicks off child Build Workflows

• Child workflow

– A workflow started from another workflow

– Runs independently with its own history

– Invocation similar to activities

– Factors functionality into reusable components

• Flow and SWF can run Child workflows and activities in parallel

Page 23: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Cron Workflow (parent)

Build Workflow (OS A)

Build Workflow (OS B)

Download files from S3

Run build tools

Upload build artifacts to

S3

Delete local files

Send email

Download files from S3

Run build tools

Upload build artifacts to

S3

Delete local files

Send email

Multiple builds in parallel

Page 24: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Multiple builds in parallel class CronWorkflow extend Decider entry_point :start def start(w_source_bucket, w_target_bucket, l_source_bucket, l_target_bucket) while true do workflow_client :w_client do |options| options.name="BuildWorkflow" options.task_list="win" end workflow_client :l_client do |options| options.name="BuildWorkflow" options.tasklist="linux" end create_timer(arg) do result1 = w_client.send_async :start, w_source_bucket, w_target_bucket result2 = l_client.send_async :start, l_source_bucket, l_target_bucket wait_for_all(result1, result2) end continue_as_new end end end

Page 25: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Concurrency in Ruby Flow

• Decider is single threaded

• Blocking semantics by default

• send_async for asynchronous execution

– Returns a Future

– Cedes control when waited on

– Uses fibers (requires Ruby 1.9 or better)

– Code looks similar to synchronous code

Page 26: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Continuous workflows

• SWF allows a workflow to stay open up to 1 year

• Workflow history grows over time as events get added

• Large history => latency

• Create new runs to keep history size in check

class BuildWorkflow extend Decider entry_point :start def start(source_bucket, target_bucket) while true do create_timer(3600) { start_build } continue_as_new(source_bucket, target_bucket) end end

Page 27: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Activity Worker

• Hosts activity implementation

• Polls SWF for tasks and dispatches to your code

• Uses two thread pools

– Polling

– Running activity tasks

• Activity implementation must be thread safe

activity_worker = ActivityWorker.new(swf.client, domain, task_list) activity_worker.add_activities_implementation(BuildActivities) activity_worker.start

Page 28: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Workflow Worker

• Hosts workflow implementation

• Polls SWF for tasks and dispatches to your code

• Uses a single thread pool for polling and running tasks

– Your logic should be light weight and deterministic

• Delegate heavy lifting to activities

worker = WorkflowWorker.new(swf.client, domain, task_list) worker.add_workflow_implementation_type(CronWorkflow) worker.add_workflow_implementation_type(BuildWorkflow) workflow_worker.start

Page 29: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Starting an execution

factory = workflow_factory swf, domain do |options|

options.workflow_name = “CronWorkflow"

options.execution_start_to_close_timeout = 3600

options.task_list = task_list

options.task_start_to_close_timeout = 3600

options.child_policy = :request_cancel

end

client = my_workflow_factory.get_client

workflow_execution = client.start(“sources“, “binaries”)

Page 30: Give your little scripts big wings:  Using cron in the cloud with Amazon Simple Workflow

Learn More

• Amazon SWF

– aws.amazon.com/swf

• AWS Flow Framework in Ruby

– TBD

• Application Samples

– aws.amazon.com/code/3015904745387737

• Webinar Videos

– Introduction to Amazon SWF

– Using the AWS Flow Framework

• AWS Flow Framework source on GitHub

– TBD