Give your little scripts big wings: Using cron in the cloud with Amazon Simple Workflow
-
Upload
amazon-web-services -
Category
Technology
-
view
3.603 -
download
2
description
Transcript of Give your little scripts big wings: Using cron in the cloud with Amazon Simple Workflow
Asad Jawahar
Give your little scripts big wings: Using cron in the cloud with
Amazon Simple Workflow
Senior Technical Program Manager
For applications with multiple connected steps…
• Amazon Simple Workflow provides the building blocks and a controller to
reduce the complexity of infrastructure programming and state machinery.
This helps customers focus on...
• Algorithms and logic that make their organizations unique
Many Use Cases
Cron
• Scheduled tasks
• Why use SWF for Cron?
– Failure handling
– Scale
– Lost tasks (office move, machine failure)
– OS provided cron not an option (shared hosting)
Roadmap
c c : code The implementation of
your control logic and
steps
d d : deployment worker processes,
machines, SWF
processing engine
l l : logical process model, flow
control, retry logic
etc
→ →
Key Simple Workflow concepts
Activities (and workers)
Workflows (and workers)
• Discrete steps in your application, processors
• Logical grouping for multiple activities, processors
Deciders • Control logic for workflows:
execution order, retry policies, timer logic, etc. – Decision tasks
Retained workflows • Execution history for
workflows: auditable, re-playable
Timers, signals • Scheduled actions, Workflow
“interrupt requests”
SWF Concepts
Workflow – Image processing Decider logic Activities
Download
Encode
Apply DRM Upload
Copyright?
Amazon SWF
Activity Workers
Workflow Workers
Decision Tasks
Activity Tasks
Activity Tasks
Activity Tasks
History
Responsibilities
AWS You
SWF
• Step sequence logic (“secret sauce”)
• Discrete step logic • Workflow workers • Activity workers • I/O data
• State of execution • Pending tasks • Execution history • Searching histories • Control engine
class CronWorkflow
extend Decider
entry_point :start
activity_client :activityClient do |options|
options.prefix_name = "CronActivities"
end
def start(arg)
while true do
create_timer(86400){activityClient.backup “myDB"}
end
end
end
Cron in SWF
class CronActivities
extend Activity
activity :backup
def backup(arg) # backup script end
end
Daily Build Process Ordinary cron, single point of failure
Download files from S3
Run build tools
Upload build artifacts to
S3
Delete local files
Send email
Handling Failures
Download files from S3
Run build tools
Upload build artifacts to
S3
Delete local files
Send email
Pass?
Pass?
Yes
Yes No
No
Resilient Cron
Job1
Job2
Job3
Job4
SWF
Resilient Cron
Distributed asynchronous interactions
Failure handling
Scalability Latency Auditability
• Coordinate scheduled tasks across distributed hosts
• Reliable messaging
Coordination engine in the cloud
• Stores and dispatches tasks • Delivery guarantees
• Add workers at any time • Need stateless workers • State provided by another system
Scalability
• Repository of distributed state
• Workers poll for work • Exactly once delivery
• Many types of failures • Different mitigations • Must react quickly
Fault Tolerance
• No loss of distributed state • Supports timeouts on work • Explicit failure modeling
• Need visibility into process execution
• Retain records for investigation
Audit History
• Index of executions • Play-by-play history of
execution • Retention of completed
executions
• Get work quickly • Route tasks to specific workers
Low Latency
• Long polling for work • Task routing through task
lists
Requirements
SWF provides
Introducing AWS Flow Framework (Ruby)
• Ease of use library for Amazon SWF APIs
• Uses standard Ruby constructs
• Open source
• Packaged as a gem
• In Private Beta (stay tuned for release)
Amazon
SWF
Benefits of AWS Flow Framework
• Run and load balance work on many machines with minimal changes
• Simplifies remote, long running, non-blocking steps
• Uses Amazon SWF to:
– Keep track of progress
– Triggers your code to execute next steps
• Failure handling built in
• Easily evolve your logic without adding complexity
Amazon
SWF
Hourly Build – Logical Control Flow
wait (1 hour)
copy files
run build task
upload files
delete local files
send email
if (failed)
retry up to 3 times
repeat
Download files from S3
Run build tools
Upload build artifacts to
S3
Delete local files
Send email
Build Cron Workflow – Execution Flow
Amazon SWF
Execution History
- Input data
- Download complete
- Build complete
- Upload complete
Decisions:
1. Schedule download [shared]
2. Schedule build [worker1]
3. Schedule upload [worker1]
DECIDER Makes decisions on what tasks to schedule, when, in what order
Start Workflow Execution
Your App, SWF Console or CLI
Starts execution of Cron Workflow
Worker
Worker
Long poll
Long Poll
Long Poll
Worker 2
Worker 1
Decision Tasks
Get task
Get task
1. /tmp, worker1
Return decisions
Shared
- Delete local file
- Email sent
2. Built
4. Schedule delete files [worker1]
5. Schedule email [worker2]
6. Execution complete
3. Uploaded 4. Deleted
5. Email sent
Get task
Hourly Build – Decider
class BuildWorkflow
extend Decider
entry_point :start
activity_client :client do |options|
options.prefix_name = "BuildActivities"
end
def start(source_bucket, target_bucket)
while true do
create_timer(25) { start_build(source_bucket, target_bucket)} end
end
def start_build(source, target)
begin dir = client.download(source) client.build(dir) client.upload(dir, target) ensure client.delete(dir) client.send_email(dir) end end
Task Routing
def start_build(source_bucket, target_bucket) activity_client :client do |options| options.prefix_name = "BuildActivities" end host_specific_task_list, dir = client.download bucket client.build(dir) do |options| options.default_task_list =host_specific_task_list end client.upload(dir, target_bucket) do |options| options.default_task_list =host_specific_task_list end client.delete(dir) do |options| options.default_task_list =host_specific_task_list end end end
Exponential Retry
def start_build(source_bucket, target_bucket) activity_client :client do |options| options.prefix_name = "BuildActivities" end dir = client.exponential_retry (:download, bucket) do |options| options.maximum_attempts = 3 end client.build(dir) client.exponential_retry (:upload, dir, target_bucket) do |options| options.maximum_attempts = 3 end client.exponential_retry (:delete, dir) do |options| options.maximum_attempts = 3 end end
Activities
class BuildActivities extend Activity
activity :download, :build, :upload, :delete do |options|
options.default_task_list = "list1"
options.version = "1"
options.default_task_heartbeat_timeout = "3600"
options.default_task_schedule_to_close_timeout = "30"
options.default_task_schedule_to_start_timeout = "30"
options.default_task_start_to_close_timeout = "30"
end
def download(bucket)
puts bucket
end
def build(dir)
puts dir
end
def upload(dir, bucket)
puts bucket
end
def delete(dir)
puts dir
end
end
Multiple builds in parallel
• Parent Cron workflow kicks off child Build Workflows
• Child workflow
– A workflow started from another workflow
– Runs independently with its own history
– Invocation similar to activities
– Factors functionality into reusable components
• Flow and SWF can run Child workflows and activities in parallel
Cron Workflow (parent)
Build Workflow (OS A)
Build Workflow (OS B)
Download files from S3
Run build tools
Upload build artifacts to
S3
Delete local files
Send email
Download files from S3
Run build tools
Upload build artifacts to
S3
Delete local files
Send email
Multiple builds in parallel
Multiple builds in parallel class CronWorkflow extend Decider entry_point :start def start(w_source_bucket, w_target_bucket, l_source_bucket, l_target_bucket) while true do workflow_client :w_client do |options| options.name="BuildWorkflow" options.task_list="win" end workflow_client :l_client do |options| options.name="BuildWorkflow" options.tasklist="linux" end create_timer(arg) do result1 = w_client.send_async :start, w_source_bucket, w_target_bucket result2 = l_client.send_async :start, l_source_bucket, l_target_bucket wait_for_all(result1, result2) end continue_as_new end end end
Concurrency in Ruby Flow
• Decider is single threaded
• Blocking semantics by default
• send_async for asynchronous execution
– Returns a Future
– Cedes control when waited on
– Uses fibers (requires Ruby 1.9 or better)
– Code looks similar to synchronous code
Continuous workflows
• SWF allows a workflow to stay open up to 1 year
• Workflow history grows over time as events get added
• Large history => latency
• Create new runs to keep history size in check
class BuildWorkflow extend Decider entry_point :start def start(source_bucket, target_bucket) while true do create_timer(3600) { start_build } continue_as_new(source_bucket, target_bucket) end end
Activity Worker
• Hosts activity implementation
• Polls SWF for tasks and dispatches to your code
• Uses two thread pools
– Polling
– Running activity tasks
• Activity implementation must be thread safe
activity_worker = ActivityWorker.new(swf.client, domain, task_list) activity_worker.add_activities_implementation(BuildActivities) activity_worker.start
Workflow Worker
• Hosts workflow implementation
• Polls SWF for tasks and dispatches to your code
• Uses a single thread pool for polling and running tasks
– Your logic should be light weight and deterministic
• Delegate heavy lifting to activities
worker = WorkflowWorker.new(swf.client, domain, task_list) worker.add_workflow_implementation_type(CronWorkflow) worker.add_workflow_implementation_type(BuildWorkflow) workflow_worker.start
Starting an execution
factory = workflow_factory swf, domain do |options|
options.workflow_name = “CronWorkflow"
options.execution_start_to_close_timeout = 3600
options.task_list = task_list
options.task_start_to_close_timeout = 3600
options.child_policy = :request_cancel
end
client = my_workflow_factory.get_client
workflow_execution = client.start(“sources“, “binaries”)
Learn More
• Amazon SWF
– aws.amazon.com/swf
• AWS Flow Framework in Ruby
– TBD
• Application Samples
– aws.amazon.com/code/3015904745387737
• Webinar Videos
– Introduction to Amazon SWF
– Using the AWS Flow Framework
• AWS Flow Framework source on GitHub
– TBD